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MEMORANDUM  FOR  ASSISTANT  SECRETARY  OF  THE  AIR  FORCE 

(FINANCIAL  MANAGEMENT  AND  COMPTOLLER) 

SUBJECT:  Report  on  Air  Force  Transition  of  Advanced  Technology  Programs  to 
Military  Applications  (Report  No.  D-2003-132) 


We  are  providing  this  report  for  review  and  comment.  We  considered 
management  comments  on  a  dr^  of  this  report  when  preparing  the  final  report. 

DoD  Directive  7650.3  requires  that  all  recommendations  be  resolved  promptly. 
We  request  that  management  provide  additional  corfunents  to  Recommendations  1, 3, 4, 

5,  and  6.  Additional  management  comments  should  by  received  by  October  14, 2003. 

If  possible,  please  send  management  comments  in  electronic  format  (Adobe 
Acrobat  file  only)  to  Audam@dodig.osd.mil.  Copies  of  tiie  management  comments  must 
contain  the  actud  signature  of  the  authorizing  official.  We  carmot  accept  the  /  Signed  / 
symbol  in  place  of  the  actual  signature.  If  you  arrange  to  send  classified  comments 
electronic^y,  they  must  be  sent  over  the  SECRET  Internet  Protocol  Router  Network 
(SIPRNET). 

We  appreciate  the  courtesies  extended  to  the  audit  staff.  Questions  should  be 
directed  to  Mr.  Bruce  A.  Burton  at  (703)  604-9071  (DSN  664-9071)  or  Mr.  Roger  H. 
Florence  at  (703)  604-9067  (DSN  664-9067).  See  Appendix  D  for  the  report  distribution. 
Audit  team  members  are  listed  inside  the  back  cover. 

By  direction  of  the  Deputy  Inspector  General  for  Auditing: 


Acquisition  Management  Directorate 


Office  of  the  Inspector  General  of  the  Department  of  Defense 


Report  No,  D-2003-132 

(Project  No.  D2002AB-0128) 


September  12,  2003 


Air  Force  Transition  of  Advanced  Technology 
Programs  to  Military  Applications 


Executive  Summary 


Who  Should  Read  This  Report  and  Why?  Science  and  teehnology  offieials  in  the 
Office  of  the  Secretary  of  Defense  and  the  Department  of  the  Air  Foree  should  read  this 
report  because  it  evaluates  the  Air  Foree’s  current  proeess  for  enhaneing  the  likelihood 
that  emerging  teehnology  would  reaeh  the  warfighter. 

Background.  Congress  and  DoD  officials  have  voiced  concern  that  teehnology  has  not 
quiekly  transitioned  to  the  warfighter.  In  1999,  the  Commander  of  Air  Force  Materiel 
Command  established  the  Applied  Technology  Councils  for  advanced  teehnology 
demonstrations  to  facilitate  the  transition  of  teehnology  projects  to  the  warfighter. 

The  audit  examined  30  S&T  projects  (24  advanced  teehnology  demonstrations  and 
6  eritical  experiments)  funded  from  the  advanced  teehnology  development  budget 
subeategory  of  the  FY  2003  Air  Force  researeh,  development,  test,  and  evaluation 
appropriation.  The  30  S&T  projects  were  valued  at  $123  million.  The  30  S&T  projects 
had  additional  planned  funding  of  $222  million  from  FY  2004  through  FY  2007. 

Results.  Although  the  Applied  Teehnology  Councils  created  a  General  Offieer  and 
exeeutive  level  review  for  the  advaneed  technology  demonstrations  and  Air  Foree 
Researeh  Laboratory  officials  perform  other  management  oversight  reviews,  procedures 
should  be  established  to  strengthen  coordination  for  all  advanced  teehnology 
development  funded  projeets  with  planned  teehnology  recipients.  Although  most 
technologies  had  working-level  integrated  product  teams,  the  teams  had  not  established 
charters  to  identify  roles  and  responsibilities.  Half  of  the  teehnologies  had  not 
established  a  transition  plan,  most  had  not  established  agreements  on  teehnology 
readiness  levels  and  exit  criteria  with  teehnology  reeipients,  12  of  the  working-level 
integrated  product  teams  had  not  doeumented  issues  and  aetion  items,  and  8  of  the 
13  aequisition  reeipients  had  not  identified  the  necessary  funding  for  teehnologies 
seheduled  to  transition  in  FYs  2003  and  2004.  Also,  the  performance  appraisal  proeess 
of  S&T  officials  needs  to  emphasize  teehnology  transitioning  as  a  performanee  element. 

As  a  result,  the  Air  Force  Researeh  Laboratory  had  planned  technology  investments  of 
$222  million  for  teehnologies  included  in  this  review  that  had  not  been  fully  eoordinated 
with  the  transition  recipient.  In  addition,  the  Air  Force  reeipients  had  a  $529  million 
funding  shortfall  for  transitioning  teehnologies  seheduled  for  availability  during 
FYs  2003  through  2005. 


Management  Comments  and  Audit  Response.  The  Assistant  Secretary  of  the  Air 
Force  (Acquisition)  agreed  with  the  report’s  conclusions  on  the  role  of  the  acquisition 
program  managers  and  the  need  to  define  the  relationships  and  responsibilities  between 
the  science  and  technology  managers  and  the  acquisition  program  managers.  However, 
the  Assistant  Secretary  did  not  agree  that  all  advanced  technology  developments  need  the 
same  level  of  management  controls  and  made  a  distinction  between  advanced  technology 
demonstration  programs  and  critical  experiments.  As  a  result,  the  Assistant  Secretary 
generally  nonconcurred  with  the  recommendations  pertaining  to  critical  experiments. 

See  the  Finding  section  of  the  report  for  a  discussion  of  management  comments  and  the 
Management  Comments  section  for  the  complete  text  of  the  comments. 

Management  comments  were  generally  nonresponsive  to  the  report  and  its 
recommendations.  We  believe  that  there  is  no  significant  difference  between  advanced 
technology  demonstration  programs  and  critical  experiments  because  both  use  advanced 
technology  development  funds.  The  audit  identified  improvements  that  are  needed  to 
coordinate  technology  development  with  planned  technology  recipients.  Also, 
technologies  are  being  developed  that  do  not  have  the  necessary  technology  transitioning 
funds  or  for  which  the  transition  funds  are  inadequate.  We  request  that  the  Assistant 
Secretary  of  the  Air  Force  (Acquisition)  provide  comments  on  the  final  report. 

Comments  should  be  received  by  October  14,  2003. 
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Background 


Defense  Acquisition.  The  Deputy  Secretary  of  Defense  issued  DoD  Directive 
5000.1,  “The  Defense  Acquisition  System,”  on  May  12,  2003.  The  directive 
states  that  science  and  technology  (S&T)  programs  shall  address  user  needs; 
maintain  a  broad-based  program  spanning  all  Defense-relevant  sciences  and 
technologies  to  anticipate  future  needs  and  those  not  being  pursued  by  civil  or 
commercial  communities;  preserve  long-range  research;  and  enable  rapid 
successful  transition  from  the  S&T  base  to  useful  military  products.  Advanced 
technology  shall  be  integrated  into  producible  systems  and  deployed  in  the 
shortest  time  practical.  Teaming  among  warfighters,  users,  developers,  acquirers, 
technology  experts,  industry,  testers,  budgeting  officials,  and  system  maintainers 
shall  begin  during  requirements  definition. 

Science  and  Technology  Guidance.  An  affordability  task  force  chartered  by  the 
Director  for  Defense  Research  and  Engineering  issued  a  handbook  and  the 
Deputy  Under  Secretary  of  Defense  for  Science  and  Technology  issued  a  guide  to 
the  Military  Departments  and  Defense  agencies  concerning  practices  that  they 
believed,  if  instituted,  would  assist  in  transitioning  technology.  Subsequently,  the 
Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  issued  a 
Manager’s  Guide  that  reemphasized  the  issues  on  technology  management  cited 
in  the  handbook  and  the  guide.  In  addition,  in  response  to  congressional  concerns 
that  DoD  had  not  been  successful  in  transitioning  technology,  the  Under  Secretary 
of  Defense  for  Acquisition,  Technology,  and  Logistics  issued  a  report  to  Congress 
identifying  why  technology  was  not  transitioning. 

Addressing  Affordability  in  Defense  Science  and  Technology  (S&T): 

A  Handbook  for  S&T  Managers.  In  October  1999,  the  DoD  S&T  Affordability 
Task  Lorce  issued  a  Handbook  that  stresses  the  importance  of  early  involvement 
of  all  candidate  acquisition  programs  in  advanced  technology  efforts.  The 
Handbook  states  that  early  involvement  of  advanced  technology  candidate 
acquisition  programs  in  research  development,  design,  test  planning, 
manufacturing,  training,  logistics,  financing,  and  contracting  are  essential  to 
address  key  issues  that  lock  in  a  majority  of  the  life-cycle  costs  of  programs.  The 
Handbook  states  that  management  tools  for  ensuring  effective  technology 
transitioning  include  establishing  integrated  product  teams  (IPTs),  creating  IPT 
charters,  identifying  quantitative  metrics  and  key  exit  criteria,  and  developing  a 
formal  transition  plan  that  is  officially  signed  by  the  technology  manager  and  the 
“customer”  (usually  an  acquisition  community  member).  Additional  management 
tools  include  preparing  an  approved  memorandum  of  agreement  or  understanding 
that  includes  roles  and  responsibilities  of  the  various  participants  and  a  funding 
strategy,  which  commits  the  acquisition  community  to  transition  the  technology. 

Technology  Transition  for  Affordability:  A  Guide  for  S&T  Program 
Managers.  In  April  2001,  the  Deputy  Under  Secretary  of  Defense  for  Science 
and  Technology  issued  a  Guide  to  provide  S&T  managers  with  strategies  to 
transition  technology  to  the  acquisition  community.  The  Guide  states  that  the 
transition  of  technology  should  be  timely  (get  the  technology  in  the  hands  of  the 
warfighter  as  soon  as  possible)  and  cost-effective  (provide  the  best  technology  at 
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the  lowest  possible  cost).  The  Guide  states  that  a  key  strategy  for  transitioning 
technology  is  early  coordination  between  the  S&T  project  manager  and  the 
receiving  acquisition  manager  to  promote  a  mutual  understanding  between  the 
two  parties. 

The  Guide  states  that  IPTs  should  include  the  S&T  product  manager,  the  S&T 
contractor,  the  acquisition  manager(s)  and  the  respective  contractor(s),  and  test 
and  evaluation  representatives.  An  IPT  should  be  formed  early  in  the  life  cycle  of 
a  technology’s  development  to  address  key  issues  that  can  greatly  affect  life-cycle 
cost  and  the  eventual  acceptance  and  implementation  of  the  technology.  Issues 
that  the  IPT  should  address  include  defining  and  agreeing  upon  quantifiable 
metrics,  such  as  cost,  performance,  and  schedule;  exit  criteria;  and  the  maturity  of 
the  technology  at  transition  identified  as  technology  readiness  levels  (TRLs)  (the 
various  levels  are  described  in  Appendix  B).  The  Guide  states  that  those  issues 
and  others  should  be  agreed  upon  in  formal  documentation  such  as  a 
memorandum  of  agreement  or  understanding  and  technology  transition  plans. 

Manager’s  Guide  to  Technology  Transition  in  an  Evolutionary 
Acquisition  Environment.  In  January  2003,  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics  reemphasized  the  guidance  issued  in  the 
Handbook  and  Guide  was  intended  to  be  a  source  of  information  to  promote 
collaboration  among  team  members.  It  provides  an  overview  of  the  processes, 
communities,  programs,  and  challenges  associated  with  technology  transition. 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 
Report  to  Congress.  In  June  2001,  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics  provided  a  Defense  Advanced  Research 
Projects  Agency  report  on  technology  transitioning  to  congressional  defense 
committees.  The  report  provided  Congress  with  the  results  of  a  review  of  the 
transition  of  research  to  the  Military  Departments  from  the  Defense  Advanced 
Research  Projects  Agency  and  addressed  issues  that  were  also  applicable  for  the 
Air  Force  transition  of  research  technology  to  acquisition  program  managers  and, 
ultimately,  to  the  warfighter.  The  report  stated  that  a  key  reason  for  difficult 
technology  transition  was  the  need  for  collaboration  among  three  diverse  groups: 
the  S&T  researcher,  the  acquisition  program  manager,  and  the  military  user. 
Effective  transition  requires  the  groups  to  work  together  as  a  team,  which  is 
frequently  a  difficult  issue.  In  addition,  for  a  technology  to  transition 
successfully,  the  acquisition  program  manager’s  prime  contractor  must  support 
the  technology  insertion,  and  the  technology  must  demonstrate  a  greater  return 
than  the  existing  capability. 

Air  Force  Process.  In  1997,  the  Air  Force  Research  Laboratory  (AFRL)  was 
created  through  the  consolidation  of  four  former  Air  Force  laboratories  and  the 
Air  Force  Office  of  Scientific  Research.  The  laboratory  comprises 
nine  directorates,  each  with  different  technology  disciplines,  located  throughout 
the  United  States.  In  addition,  the  Air  Force  Office  of  Scientific  Research 
manages  national  and  international  basic  research.  The  nine  technology 
directorates  are  responsible  for  exploratory  technology  development  (applied 
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research)  and  advanced  technology  development  research  areas,  as  well  as  other 
research  funded  with  non-AFRL  funds.  The  subject  of  this  audit  was  research 
projects  funded  with  AFRL  advanced  technology  development  funds. 

The  AFRL  Science  and  Technology  Mission  Area  Strategic  Plan  for  FYs  2002 
through  2009  provide  a  roadmap  for  the  future  technologies.  The  objective  of  the 
S&T  program  is  to  provide  the  technical  foundation  for  the  Air  Force.  In 
addition,  the  AFRL  Center  Strategic  Plan  for  FY s  2002  through  2009  established 
a  core  strategy  objective  to  demonstrate  and  transition  technology.  To  accomplish 
that  objective,  AFRL  partnered  with  the  Major  Commands  (MAJCOMs)  to  ensure 
that  at  least  50  percent  of  the  AFRL  advanced  technology  development  funds  are 
directed  towards  advanced  technology  demonstrations  (ATDs)  through  FY  2009. 
Also,  AFRL  partners  with  the  MAJCOMs  and  the  Air  Force  Materiel  Command 
product  centers  to  ensure  that  75  percent  of  the  transition  plans  for  ATDs  are 
funded. 

Objectives 


The  audit  objective  was  to  determine  whether  the  Air  Force  was  successful  in 
transitioning  advanced  technology  projects  to  military  applications.  Specifically, 
we  determined  whether  the  Air  Force  had  established  a  process  to  successfully 
transition  technology.  We  also  evaluated  management  controls  at  AFRL  as  they 
relate  to  the  audit  objective.  See  Appendix  A  for  a  discussion  of  the  audit  scope 
and  methodology,  the  review  of  the  management  control  program,  and  prior 
coverage  related  to  the  audit  objectives. 

The  audit  examined  30  S&T  projects  (24  ATDs  and  6  Critical  Experiments  (CEs)) 
funded  from  the  advanced  technology  development  budget  subcategory  of  the 
EY  2003  Air  Eorce  research,  development,  test,  and  evaluation  appropriation. 

The  30  S&T  projects  were  valued  at  $123  million.  The  30  S&T  projects  had 
additional  planned  funding  of  $222  million  from  EY  2004  through  EY  2007.  The 
audit  examined  whether  working-level  IPTs  were  established,  whether  they 
included  the  planned  recipients,  whether  charters  were  established  for  the  IPTs, 
whether  official  memorandums  of  understanding  or  agreement  or  technology 
transition  plans  had  been  established,  and  whether  agreements  included  TRLs  and 
exit  criteria.  The  audit  did  not  review  exploratory  research  (applied  research)  or 
AERE  projects  conducted  with  non-AERL  funding. 
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Air  Force  Science  and  Technology 
Process 

In  2001,  the  Commander  of  the  Air  Foree  Materiel  Command  and  the 
Prineipal  Deputy  Assistant  Secretary  of  the  Air  Force  (Acquisition  and 
Management)  established  interim  guidance  for  the  Applied  Technology 
Councils  (ATCs)  to  facilitate  the  technology  transition  of  advanced 
technology  demonstrations  to  the  warfighter.  Although  the  ATCs  created 
a  General  Officer  and  executive  level  of  review  for  the  ATDs  and  other 
management  oversight  processes,  improvements  in  coordination  are  still 
needed  in  the  AFRL  at  the  acquisition  program  manager  level. 

Specifically,  although  23  technologies  did  have  working-level  IPTs,  other 
critical  elements  for  transitioning  were  missing: 

•  All  but  two  working-level  IPTs  had  not  established  charters  to 
identify  roles  and  responsibilities, 

•  Half  of  the  ATDs  had  not  established  technology  transition  plans 
for  emerging  technologies, 

•  Most  of  the  technologies  did  not  have  established  agreements  on 
TRLs  and  exit  criteria  with  technology  recipients, 

•  Twelve  of  the  working-level  IPTs  did  not  document  issues  and 
action  items  to  prevent  development  problems  and  provide 
accountability,  and 

•  Eight  of  the  13  acquisition  recipients  for  technologies  scheduled  to 
transition  in  FYs  2003  and  2004  did  not  identify  funding. 

In  addition,  the  performance  appraisal  process  for  S&T  officials  needs 
improvement  to  emphasize  technology  transitioning  as  a  performance 
element. 

These  conditions  exist  because  Air  Force  S&T  management  had  not  fully 
implemented  the  best  practices  advocated  by  the  Deputy  Under  Secretary 
of  Defense  for  Science  and  Technology.  The  status  of  formal  agreements 
between  the  technology  and  recipient  communities  is  not  a  subject  at  the 
ATCs.  The  AFRL  had  not  established  a  requirement  for  formal 
coordination  on  critical  experiment  (non-ATD)  research  with  the  planned 
technology  recipient  and  also  had  not  established  minimum  documentation 
standards  for  technology  development  to  include  minutes  of  IPTs,  record 
of  action  items,  and  decisions.  As  a  result,  the  AFRL  planned  technology 
investments  of  $222  million  for  technologies  included  in  this  review  had 
not  been  fully  coordinated  with  the  transition  recipient.  In  addition,  the 
Air  Force  technology  recipients  had  a  $529  million  transition-funding 
shortfall  for  all  AFRL  emerging  technologies  scheduled  to  transition  from 
LYs  2003  through  2005. 
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Air  Force  Advance  Technology  Program 


Air  Force  guidance  for  planning  and  managing  S&T  projects  was  created  before 
the  AFRL  was  established  in  1997  and  does  not  describe  the  current  process  used 
to  plan  and  manage  S&T  efforts.  Since  1997,  the  AFRL  organizational  structure 
and  the  Air  Force  S&T  process  have  changed  and,  on  June  13,  2003,  the  Secretary 
of  the  Air  Force  revised  Air  Force  Policy  Directive  61-1,  Management  of  Science 
and  Technology,  which  provides  overarching  S&T  guidance  to  reflect  those 
changes.  The  guidance  updates  the  roles  and  responsibilities  of  all  Air  Force 
organizations  directly  involved  in  managing  S&T.  Three  key  components  of  the 
Air  Force  include  the  ATCs,  baseline  reviews,  and  the  Acquisition  Center  of 
Excellence  processes. 

Applied  Technology  Councils.  In  2001,  the  Commander,  Air  Force  Materiel 
Command  and  the  Principal  Deputy  Assistant  Secretary  of  the  Air  Force 
(Acquisition  and  Management)  established  interim  guidance  for  the  ATCs.  The 
ATCs  were  designed  to  establish  a  closer  link  between  the  AFRL;  product, 
logistical,  and  test  centers  of  the  Air  Force  Materiel  Command;  and  the 
MAJCOMs.  The  Air  Force  Materiel  Command  product  centers  identify  required 
transition  funding,  prepare  technology  transition  schedules  for  acquisition 
programs  receiving  ATD  technologies,  and  recommend  candidate  ATD  priorities. 

The  objective  of  the  ATCs  is  to  review  and  approve  technology,  and  prioritize 
technology  funding.  The  ATCs  provide  a  senior-level  forum  to  facilitate 
transition  of  technology  from  AFRL  into  advanced  system  developments  and 
fielded  system  upgrades.  In  addition,  the  ATCs  are  to  provide  MAJCOM 
customer  focus  for  development  efforts  in  AFRL  based  on  warfighter 
requirements  to  improve  Air  Force  warfighting  capabilities.  The  ATC  process 
was  established  in  November  1999. 

Each  ATC  semiannually  reviews  the  status  of  existing  approved  ATDs,  approves 
(commissions)  advancing  S&T  efforts  as  ATDs,  and  offers  guidance  to  AERE  on 
future  candidate  ATD  programs.  The  ATC  process  classifies  ATDs  in  the 
following  categories: 

•  Category  1 :  MAJCOM  supports  and  has  programmed  required  funding 
for  transitioning  to  an  acquisition  effort  within  the  Euture  Years  Defense 
Plan. 

•  Category  2A:  MAJCOM  supports  and  is  committed  to  identifying 
transition  funding  in  the  next  budgeting  cycle. 

•  Category  2B:  MAJCOM  supports  but  is  not  currently  able  to  budget  for 
the  transition. 

•  Category  3:  Warfighter  does  not  support  and  has  no  plan  to  use  the 
technology. 
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Baseline  Reviews.  In  December  2001,  the  AFRL  began  baseline  reviews  as  part 
of  an  internal  management  control  process.  Baseline  reviews  are  conducted  on 
ATDs  and  high  visibility  programs  and  establish  agreements  between  AFRL 
technology  directorates  and  AFRL  Command  Section  officials  on  the  developing 
technology.  ATD  baseline  reviews  are  conducted  semiannually  to  advise  AFRL 
executive  officials  on  the  progress  of  the  technology  development  efforts. 

Baseline  reviews  are  intended  to  provide  accountability  on  the  cost,  schedule, 
program,  and  risk  for  developing  technologies.  Also,  baseline  reviews  ensure  that 
resources  remain  committed  to  the  research  efforts,  agreements  with  customers 
are  met,  and  problems  are  identified  early. 

Acquisition  Center  of  Excellence.  In  June  2002,  the  AFRL  established  the 
Acquisition  Center  of  Excellence  to  increase  the  effectiveness  of  the  AFRL 
functions,  processes,  and  efforts.  The  mission  of  the  Acquisition  Center  of 
Excellence  program  is  to  work  in  partnership  with  AERL  and  customer 
organizations  to  increase  the  speed  at  which  required,  capable,  and  affordable 
technology  transitions  to  the  warfighter. 

The  initial  goals  of  the  Acquisition  Center  of  Excellence  are  to  improve  the  speed 
and  ease  the  delivery  of  new  technology  capabilities  into  Air  Eorce  weapon 
systems,  to  define  and  implement  a  comprehensive  framework  for  laboratory 
technology  transitions,  and  to  investigate  strategic  collaboration  initiatives  with 
industry.  An  Acquisition  Center  of  Excellence  vision  includes  the  involvement  of 
the  technology  customers  on  S&T  program  IPTs  in  developing  the  requirements 
and  exit  criteria  and  participating  in  key  program  decisions.  The  AERL 
Acquisition  Center  of  Excellence  also  adopted  the  Transformational  Management 
for  Accelerated  Technology  Transition  process  that  applies  systems  engineering 
principles  across  AERL  research  programs. 

As  part  of  the  Acquisition  Center  of  Excellence  program,  the  AERL  initiated  a 
management  process  called  the  Integrated  Product  and  Process  Development  that 
strives  to  provide  a  disciplined  systems  engineering  approach  to  enhance 
technology  development  and  coordination.  Integrated  Product  and  Process 
Development  involves  thinking  ahead  and  balancing  performance,  producibility, 
cost,  schedule,  and  risk— all  integral  elements  of  the  S&T  effort. 

Critical  Experiments  Guidance.  CEs  are  advance  technology  development- 
funded  efforts  that  have  not  been  submitted  to  the  ATCs  as  ATD  proposals.  Air 
Eorce  guidance  provides  limited  policy  and  procedures  on  managing  CEs,  even 
though  these  experiments  represent  about  $380  million  or  53  percent  of  the  AERL 
advanced  technology  development  funding  for  EY  2003.  Air  Eorce 
Instruction  61-105,  “Planning  for  Science  and  Technology,”  July  22,  1994, 
requires  that  the  MAJCOMs  review  advanced  development  efforts  for  relevancy 
to  future  technology  needs.  The  reviews  include  selected  CEs  and  may  prioritize 
the  selected  CEs  based  on  future  technology  needs.  The  ATC  process  reviews 
only  CEs  that  are  candidates  for  ATDs. 
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Technology  Development  Guidance 


The  interim  guidance  issued  by  the  Deputy  Secretary  of  Defense  states  that  S&T 
programs  shall  address  user  needs,  maintain  a  broad-based  program  spanning  all 
Defense-relevant  sciences  and  technologies  to  anticipate  future  needs  and  those 
not  being  pursued  by  civil  or  commercial  communities,  preserve  long-range 
research,  and  enable  rapid  successful  transition  from  the  S&T  base  to  useful 
military  products.  The  Handbook  provides  elements  of  best  practices  and 
procedures  captured  in  the  form  of  criteria  for  S&T  managers.  Those  criteria 
were  endorsed  by  the  Deputy  Under  Secretary  of  Defense  for  Acquisition  and 
Technology’s  Affordability  Task  Force  that  was  charted  by  the  Director  for 
Defense  Research  and  Engineering  to  develop  solutions  on  how  to  strengthen 
DoD  S&T  programs. 

Air  Force  Science  and  Technology  Products  Reviewed 


The  AFRL  needs  to  strengthen  its  S&T  coordination  with  program  recipients. 
The  Handbook  and  Guide  recommend  establishing  memorandums  of  agreement 
or  understanding  with  the  acquisition  program  manager(s),  including  agreement 
on  TRLs  and  exit  criteria,  to  ensure  that  acquisition  programs  have  the  necessary 
funding  for  S&T  integration.  In  addition,  the  AFRL  should  require  S&T  product 
managers  to  document  IPT  issues  to  prevent  potential  development  problems 
resulting  from  key  personnel  changes. 

The  audit  examined  30  S&T  projects  (24  ATDs  and  6  CEs)  funded  from  the 
advanced  technology  development  budget  subcategory  of  the  FY  2003  Air  Force 
research,  development,  test,  and  evaluation  appropriation.  The  30  S&T  projects 
were  valued  at  $123  million.  The  30  S&T  projects  had  additional  planned 
funding  of  $222  million  from  FY  2004  through  FY  2007.  Twenty-two  of  the 
24  ATDs  (two  completed  ATDs  transitioned  to  other  ATDs)  had  23  planned 
recipient  acquisition  programs  and  the  6  CEs  had  6  planned  recipients.  The 
following  table  summarizes  the  audit  results  by  ATD  and  CE,  using  the 
recommended  best  business  practices  in  the  Handbook  and  Guide. 
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Table  1.  Science  and  Technology  Advanced  Technology 
Demonstrations  and  Critical  Experiments  Examined 

(ratio  shows  positive  responses  to  total  examined)^ 


2 

Number  of  Oeeurrenees  Pereent  of  Oecurrenees 


24  ATDs 

6  CEs 

24  ATDs 

6  CEs 

IPTs  at  the  product  level 

Team  established 

19  of  22 

4  of  6 

86 

67 

Charter  established 

1  of  19^ 

lof4^ 

5 

25 

Acquisition  program 
manager  included 

19  of 20^ 

4  of  4 

95 

100 

Acquisition  program 
prime  contractor 
included 

18  of  19^ 

3  of  3' 

95 

100 

Acquisition 

Program  Manager 

MOA/MOU/TTP^ 

11  of  22 

1  of  6 

50 

17 

Exit  TRLs  formally 
agreed 

7  of  23^ 

Oof  6 

30 

0 

Exit  criteria  formally 
agreed 

10  of 23^ 

1  of  6 

43 

17 

Eunding  for  EYs  2003 
and  2004  transitions  by 
the  acquisition  recipient 

5  of  13^ 

Oof  o’ 

38 

Note:  Appendix  C  provides  the  supporting  detail  to  the  table  by  ATD  and  CE. 


'Draft  documents  were  not  considered  a  positive  response.  Response  deemed  “not  applicable”  were  not 
included  in  the  base. 

^The  percentage  of  occurrences  represent  the  results  of  projects  examined  and  may  not  necessarily 
represent  the  results  of  all  projects  in  the  AFRL  advanced  technology  demonstration  funded  universe. 

^Nineteen  of  the  24  ATDs  reviewed  had  working-level  IPTs.  The  base  for  the  CEs  was  reduced  by  1 
because  1  project  did  not  have  an  identified  acquisition  program  contractor. 

"'Four  of  the  six  CEs  had  working-level  IPTs. 

^One  technology  had  two  identified  users,  therefore  the  base  was  increased  by  one. 

'’Memorandum  of  Agreement,  Memorandum  of  Understanding,  Technology  Transition  Plan. 

^Twelve  of  the  24  ATDs  had  near-term  transitions  dates  (One  of  the  ATDs  has  two  recipients,  therefore  the 
12  is  13)  .  AFRL  does  not  require  CEs  to  have  a  transition  path  and  therefore  the  CEs  do  not  have  an 
established  scheduled  to  transition. 
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Integrated  Product  Teams 


The  Deputy  Secretary  of  Defense  issued  interim  acquisition  guidance  on 
October  30,  2002,  which  states  that  IPTs  will  be  used  and  that  they  will  include 
warfighters,  users,  developers,  acquirers,  technology  experts,  industry,  testers, 
budgeting  officials,  and  system  maintainers.  In  addition,  the  Handbook  and 
Guide  cite  best  business  practices,  including  establishment  of  working-level  IPTs 
and  IPT  charters.  To  be  effective,  IPTs  must  include  the  acquisition  program 
manager(s)  and  have  an  established  charter.  The  prime  contractor  should  also  be 
considered  for  IPT  participation,  if  appropriate,  to  facilitate  the  technology 
integration. 

Integrated  Product  Teams  Established.  The  Handbook  and  Guide 
recommended  the  establishment  of  IPTs  at  the  working  level  for  all  the  S&T 
efforts  and  ATDs.  AFRL  officials  established  working-level  IPTs  for  19  of  the 
22  ATDs  and  4  of  the  6  CEs.  Nineteen  of  the  20  planned  technology  recipients 
were  participating  in  the  19  working-level  IPTs  to  some  degree  (1  technology  had 
2  planned  recipients).  Four  of  the  six  CEs  had  established  a  working-level  IPT 
and  all  four  CEs  had  participation  by  the  planned  recipient. 

Integrated  Product  Team  Charters.  The  Handbook  recommends  the 
establishment  of  IPT  charters.  The  Handbook  states  that  IPT  charters  provide  the 
best  way  to  minimize  team  misunderstanding.  The  Handbook  provides  that  each 
charter  should  include: 

•  The  mission  and  objectives  of  the  team, 

•  The  metrics  to  evaluate  the  team’s  progress, 

•  The  scope  of  the  team’s  responsibility, 

•  The  relationship  of  the  team  with  other  teams, 

•  The  authority  and  accountability  of  the  team, 

•  The  resources  available  for  the  team,  and 

•  A  team  membership  list. 

Only  1  of  the  19  ATDs  and  1  of  the  4  CEs  had  established  some  form  of  a  charter 
for  their  IPTs.  The  AFRL  process  does  not  require  the  establishment  of  a  charter, 
and  AFRL  officials  stated  that  other  required  research  supporting  documentation 
for  baseline  and  ATC  reviews  capture  many  of  the  elements  that  IPT  charters 
would  contain.  However,  our  review  identified  a  general  lack  of  IPT 
documentation  of  meeting  results,  action  items,  and  issues  needing  continued 
examination,  posing  a  risk  in  orderly  continuation  of  S&T  efforts.  This  risk  is 
avoidable  by  requiring  the  documentation  of  significant  program  decisions  and 
development  issues  and  providing  for  accountability.  The  AFRL  does  not  require 
the  establishment  of  IPT  charters  to  document  the  roles  and  responsibilities  of 
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AFRL  or  acquisition  officials  or  require  the  establishment  of  a  means  to  control 
and  track  issues,  deeisions,  assignments,  or  action  items  resulting  from  IPX 
meetings.  The  lack  of  documentation  also  inhibits  aeeountability. 

Acquisition  Program  Prime  Contractor,  One  goal  of  the  Air  Force  S&T 
program  is  to  transition  teehnology  to  an  aequisition  program  and,  subsequently, 
to  the  warfighter.  The  Guide  and  Handbook  reeommend  that  the  technology 
receiver’s  prime  eontraetor  participate  in  the  IPT  to  facilitate  the  integration  of  the 
evolving  teehnology  into  the  reeeiving  platform.  Of  the  19  ATDs  that  established 
IPTs,  18  planned  reeipients  had  prime  eontractors  partieipating. 

Of  the  four  CEs  that  established  IPTs,  three  recipient  programs  had  prime 
contractors  participating  in  the  working-level  IPT.  The  remaining  CE  teehnology 
reeipient  had  not  identified  a  prime  contractor.  Air  Eorce  had  no  official  S&T 
guidance  on  prime  contractor  representation  in  IPTs  for  ATD  or  CE  efforts. 

Role  of  Acquisition  Program  Managers 


To  improve  the  likelihood  of  technology  transition,  acquisition  program  managers 
must  make  a  firm  eommitment  to  transition  the  teehnology  to  their  programs.  The 
commitment  should  inelude  a  formal  and  up-to-date  memorandum  of  agreement, 
understanding,  or  teehnology  transition  plan  (TIP)  between  the  S&T  product 
manager  and  the  aequisition  program  manager(s).  Eaeh  memorandum  should 
speeify  the  relationships  and  responsibilities  of  the  S&T  product  manager  and  the 
receiving  aequisition  program  manager(s).  The  agreement  should  address  system 
requirements,  funding,  personnel  support,  exit  criteria,  and  TRLs.  Within  the 
AERE,  TTPs  represent  and  funetion  as  the  memorandum  of  agreement  or 
understanding.  TTPs  represent  agreement  between  the  S&T  and  planned 
recipient  communities  on  TRLs  and  exit  eriteria.  Our  review  showed  that  TTPs 
do  not  always  include  agreement  on  TRLs  and  exit  eriteria.  The  AERE  draft 
guidance  requires  the  establishment  of  TTPs  that  include  agreement  on  TRLs  and 
exit  eriteria.  The  establishment  of  TTPs  that  inelude  TRLs  and  exit  eriteria  is  one 
of  the  subjects  presented  at  the  baseline  reviews. 

Technology  Transition  Plans,  Air  Eorce  Materiel  Command  Instruction  61-102, 
“Technology  Transition  Planning,”  March  13,  1995,  requires  the  development  of 
TTPs.  The  instruetion  applies  only  to  ATDs  and  states  that  TTPs  will  represent 
agreement  between  the  AERE  and  the  reeeiving  aequisition  program  manager(s) 
on  the  development  of  the  teehnology  for  transitioning  and  the  exit  criteria. 

Eleven  of  the  22  ATDs  had  established  TTPs.  Of  those  1 1  TTPs,  6  were  current, 

3  were  being  revised,  and  2  had  not  been  updated  within  the  last  3  years  (1  was 
7  years  old). 

The  six  CEs  had  six  planned  reeipients.  Only  one  reeipient  had  established  a  TTP 
and  that  TTP  was  4  years  old.  AERE  does  not  require  the  establishment  of  TTPs 
for  CEs.  Although  CEs  are  related  to  early  stages  of  advaneed  technology 
development,  the  establishment  of  the  substanee  of  a  TTP  should  be  a 
requirement  to  ensure  that  the  potential  planned  recipient  is  aware  of  the 


10 


technology  effort  and  that  some  form  of  coordination  and  agreement  on  the 
technology’s  development,  requirements,  and  exit  criteria  are  established  for  a 
baseline  for  technology  development. 

Technology  Readiness  Levels.  DoD  adopted  TRLs  in  response  to  a  General 
Accounting  Office  Report,  “Best  Practices:  Better  Management  of  Technology 
Development  Can  Improve  Weapon  System  Outcomes,”  issued  in  July  1999.  A 
TRL  is  an  assessment  of  the  technical  maturity  of  an  S&T  effort.  In  July  2001, 
the  Deputy  Under  Secretary  of  Defense  for  Science  and  Technology  issued  a 
memorandum  to  the  Military  Departments  and  Defense  agencies  that  emphasized 
the  development  of  TRLs.  The  TRLs  range  from  one  through  nine,  with  more 
mature  S&T  efforts  having  a  higher  TRL  and  a  lower  risk  for  the  acquisition 
program.  Only  7  of  23  recipients  had  agreed  to  TRLs.  None  of  the  six  CEs  had 
agreements  on  TRLs. 

Exit  Criteria,  The  exit  criteria  for  each  S&T  effort  describe  the  current 
capabilities,  the  expected  performance  parameters  and  conditions  of 
measurement,  the  range  of  acceptable  performance  improvements,  and  the  test 
conditions  and  verification  methods  for  measuring  performance.  The  S&T 
manager  and  the  acquisition  program  managers,  in  collaboration  with  the  IPT, 
develop  exit  criteria  that  are  appropriate  for  transitioning  the  technology.  The 
22  ATDs  had  23  technology  recipients.  Agreements  on  exit  criteria  were 
established  for  10  of  these  23  recipients.  Of  the  six  CEs,  only  one  recipient  had 
agreed  to  exit  criteria. 

Acquisition  Program  Funding 


The  DoD  research,  development,  test,  and  evaluation  budget  is  divided  into 
seven  budget  activities.  The  S&T  community  receives  funding  from  the  first 
three  budget  activities  only:  basic  research,  applied  research,  and  advanced 
technology  development.  The  acquisition  community  is  funded  with  three  budget 
activities:  demonstration  and  validation,  engineering  and  manufacturing 
development,  and  operational  systems  development.  A  seventh  budget  activity, 
management  support,  is  directed  toward  installations  or  operations  required  for 
general  research  and  development  use.  This  separation  of  research,  development, 
test,  and  evaluation  funding  between  the  S&T  and  acquisition  communities  and 
the  shrinking  of  the  research,  development,  testing,  and  evaluation  budget  make 
coordination  between  the  S&T  program  managers  and  acquisition  program 
managers  very  critical. 

Within  the  Air  Eorce,  the  operational  MAJCOMs  prioritize  the  activity  funds 
budgeted  for  the  acquisition  community.  In  this  prioritization  process,  the 
MAJCOMs  determine  the  funds  planned  for  the  technology  transition  from  the 
S&T  community  to  the  acquisition  community.  As  part  of  the  review,  we 
examined  the  transitioning  fund  requirements  for  the  technologies  reviewed  in  the 
audit  and  for  the  47  technologies  scheduled  to  transition  between  EYs  2003  and 
2005  that  are  under  development  at  AERL.  The  review  found  significant 
technology  transitioning  funding  shortfalls. 
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Funding  for  ATDs  Reviewed.  The  MAJCOMs  and  acquisition  program 
managers  were  not  providing  the  funding  required  for  technology  transitioning. 
Twelve  of  the  22  ATDs  were  scheduled  to  transition  to  acquisition  program 
managers  in  FYs  2003  or  FY  2004.  These  12  ATDs  had  13  acquisition  recipients. 
Of  these  13,  only  5  acquisition  programs  had  identifiable  funds  for  the 
transitioning  technologies.  Acquisition  programs  should  be  required  to 
specifically  identify  funds,  particularly  for  near-term  transitions,  to  ensure  that 
funds  are  available  as  a  requirement  for  continued  research  expenditure  by  AFRL. 
Without  adequate  funding  for  technology  transition,  AFRL  will  not  be  able  to 
determine  whether  continued  investment  in  ATDs  is  beneficial  or  whether  limited 
research  funds  of  AFRL  should  be  directed  to  other  promising  technology 
transition  candidates. 

Funding  Requirements  for  All  Near-Term  Transitions,  The  AFRL  plans  to 
have  47  of  the  62  ATDs  ready  for  transitioning  between  FYs  2003  and  2005.  The 
MAJCOMs  identified  a  transition  funding  requirement  of  $2,406.8  million  for 
45  of  47  ATDs;  however,  only  $1,877.4  million  has  been  budgeted  for  FYs  2003 
through  2007,  resulting  in  a  research,  development,  test,  and  evaluation  shortfall 
of  $529.4  million  for  FYs  2003  through  2007,  if  the  technology  development  is 
successful.  The  MAJCOMS  had  not  identified  the  funding  requirements  for  the 
remaining  two  ATDs  and  only  partially  identified  the  funding  requirements  for  5 
of  the  ATDs  in  Table  2.  Table  2  shows  the  FYs  2003  through  2007  funding 
required,  budgeted,  and  shortfalls  for  ATDs  that  are  planned  to  become  available 
in  FYs  2003  through  2005. 

Table  2,  Summary  of  Near-Term  ATD  Categories 
By  Technology  Availability  Year 

(Dollars  are  in  millions) 


Fiscal  Year  of 

Number  1 - 

-FYs  2003  -  2007- 

Technology 

Availabilitv 

of 

ATDs 

Funding 

Required 

Funding 

Budgeted 

Shortfall 

2003 

20 

$1508.3 

$1376.0 

$132.3 

2004 

11 

498.8 

239.0 

259.8 

2005 

14 

399.7 

262.4 

137.3 

Total 

45 

$2,406.8 

$1,877.4 

$529.4 

Table  2  does  not  include  transition  funding  that  is  embedded  in  the  acquisition, 
such  as  preplanned  product  improvements. 

In  addition  to  the  above,  the  Air  Force  also  used  Warfighter  Rapid  Acquisition 
Process  (known  as  WRAP)  funds  to  transition  S&T  efforts.  Warfighter  Rapid 
Acquisition  Process  funds  were  used  to  transition  two  ATDs  in  FYs  2002  and 
2003.  The  Air  Force  had  not  budgeted  for  Warfighter  Rapid  Acquisition  Process 
funds  in  the  President’s  Budget  for  FY  2004  and  subsequent  years;  therefore,  this 
source  of  transition  funds  will  not  be  available. 
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Personnel  Performance  Assessments 


The  AFRL  established  a  performance  element  in  the  S&T  Statements  of  Duties 
and  Experience  that  addresses  technology  transitioning.  The  S&T  duties  state  that 
the  S&T  official  will  seek  opportunities  for  technology  transitioning,  lead  or  serve 
as  a  key  technology  member  of  teams  for  transitioning,  and  organize  and  market 
overall  technology  transitions  at  senior  management  levels.  However,  the  S&T 
duties  do  not  specifically  include  the  requirements  identified  in  the  Handbook  and 
Guide.  For  example,  the  AFRL  S&T  duties  do  not  require  the  need  for  each  S&T 
advanced  technology  development- funded  effort  to  establish  working-level  IPTs 
with  all  planned  recipients,  to  create  IPT  charters,  to  coordinate  and  agree  upon 
quantitative  metrics  and  key  exit  criteria  with  all  planned  recipients,  to  develop 
TTPs,  and  to  develop  and  maintain  up-to-date  memorandums  of  agreement  or 
understanding.  The  inclusion  of  these  requirements  in  S&T  duties  would  be  a 
better  measure  of  S&T  personnel  performance  and  would  enhance  the  likelihood 
of  technology  transition. 

Recommendations,  Management  Comments,  and  Audit 
Response 


We  recommend  that  the  Commander,  Air  Force  Research  Laboratory: 

1,  Require  the  establishment  of  integrated  product  teams  and 
charters  for  all  advanced  technology  development  efforts  to  include 
representatives  from  the  Air  Force  Major  Commands,  the  candidate 
acquisition  program  office(s),  and  the  acquisition  program  office  prime 
contractor(s),  where  applicable. 

Management  Comments,  The  Assistant  Secretary  of  the  Air  Force  (Acquisition) 
nonconcurred  with  the  recommendation.  The  Assistant  Secretary  stated  that 
integrated  product  teams  (IPTs)  and  charters  should  not  be  required  for  all 
advanced  technology  development  and  that  if  the  requirement  was  limited  to 
advanced  technology  demonstration  (ATD)  programs,  the  Air  Force  would 
concur.  He  stated  that  non-advanced  technology  demonstration  programs  (known 
as  Critical  Experiments  (CEs))  are  not  sufficiently  technically  mature  for  an 
objective  evaluation  by  the  potential  receiver  of  the  technology  and  that  such  a 
review  would  require  a  massive  amount  of  resources  because  there  are 
approximately  300  CEs  compared  to  approximately  60  ATDs.  The  Assistant 
Secretary  stated  that  IPTs  are  appropriate  for  ATDs  and  encourages  IPTs  for  CEs; 
however,  he  believes  that  requiring  IPTs  for  CEs  would  increase  the  AFRL 
workload  five-fold. 

The  Assistant  Secretary  stated  that  AFRL  had  established  a  format  for  IPT 
charters. 

Audit  Response,  The  Assistant  Secretary’s  comments  are  not  fully  responsive  to 
the  recommendation.  The  Assistant  Secretary  stated  that  he  would  concur  with 
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this  recommendation  if  it  applied  only  to  ATD  programs  because  CEs  are  not 
sufficiently  technically  mature.  We  do  not  agree  with  the  Air  Force  in  making  a 
distinction  between  ATDs  and  CEs  because  CEs  are  S&T  projects  that  also  use 
advanced  technology  development  funds  under  the  research,  development,  test, 
and  evaluation  appropriation,  and,  as  such,  should  be  held  to  a  similar  level  of 
development  coordination  as  ATDs.  The  DoD  Financial  Management 
Regulation,  June  2002,  defines  advanced  technology  development  under  the 
research,  development,  test  and  evaluation  appropriation  as  efforts  that  are  proof 
of  technological  feasibility  and  assessment  that  have  a  direct  relevance  to 
identified  military  needs.  CEs  comprise  53  percent  ($380  million)  of  the  Air 
Force’s  advanced  technology  development  budget  for  FY  2003  and  should  have 
specific,  measurable,  major  technological  advancements  to  be  achieved.  Projects 
under  this  category  should  be  directly  related  to  identified  military  needs,  and 
should  demonstrate  their  general  military  use.  As  such,  early  coordination  with 
the  potential  recipient  is  necessary  to  ensure  their  military  relevance  and  to 
enhance  the  transition  of  the  CE  technology. 

Early  coordination  with  potential  recipients  is  emphasized  not  only  in  the 
Handbook  and  Guide,  but  also  in  the  Defense  Systems  Management  College 
training  course,  “Program  Management  for  S&T  Manager,”  STM  301,  formally 
known  as  “Technology  Insertion  in  Defense  Systems  Acquisitions.”  The 
Handbook,  Guide,  and  the  training  course  do  not  provide  lesser  coordination  for 
any  S&T  effort  that  is  funded  by  advanced  technology  development 
appropriations.  With  the  emphasis  on  transitioning  technology  from  the  S&T  to 
the  potential  recipient,  we  believe  that  the  recommendation  remains  appropriate 
and  has  merit. 

In  preparing  this  report,  we  reviewed  the  AFRL  ATD  database  and  found  two 
files  that  contained  listings  of  CEs.  The  first  file  identified  107  CEs.  Further 
examination  of  this  CE  file  showed  that  29  of  the  107  CEs  had  the  technology 
available  in  FY  2003  or  before.  Of  the  remaining  78  CEs,  27  were  to  have 
technology  available  during  FYs  2004  or  2005:  Accordingly,  in  the  near-term,  at 
least  27  CEs  should  be  coordinated  through  IPTs  with  the  planned  technology 
receiver. 

The  second  file  provides  another  listing  of  250  CEs  by  fiscal  year,  of  which  129 
were  funded  in  FY  2003  and  subsequent  years,  while  another  25  were  to  begin 
funding  in  FY  2004.  The  remaining  96  CEs  were  either  completed  prior  to  FY 
2003  or  had  no  funding  in  FYs  2003  and  2004.  This  listing  provides  a  funding 
profile  for  CEs  only,  and  does  not  have  a  technology  description. 

We  request  the  Assistant  Secretary  to  reconsider  his  position  and  provide 
additional  comments  to  the  final  report. 

2,  Establish  procedures  that  require  the  status  of  technology 
transition  plans  between  the  science  and  technology  manager,  the  Air  Force 
Major  Commands,  and  the  candidate  acquisition  program  offices  be  a 
subject  of  review  at  the  Applied  Technology  Councils. 
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Management  Comments.  The  Assistant  Secretary  concurred  and  stated  that  the 
AFRL  established  procedures,  on  June  23,  2003,  requiring  the  Applied 
Technology  Councils  to  review  the  status  of  technology  transition  plans.  The 
procedures  will  be  formalized  in  an  Air  Force  instruction  by  March  2004. 

3.  Review  the  technology  paths  or  plans  for  the  eight  unfunded 
science  and  technology  efforts  that  are  scheduled  to  transition  in  FYs  2003 
and  2004,  and  discontinue  development  for  those  that  do  not  have  formal 
acquisition  program  support  with  identified  funding. 

Management  Comments,  The  Assistant  Secretary  nonconcurred  with  the 
recommendation,  citing  that  the  ATC  process  was  established  with  different 
technology  categories  to  allow  both  technology  push  and  pull.  The  technology 
balance  of  push  and  pull  is  required  to  maintain  technological  superiority.  He 
said  that  eliminating  developing  technologies  that  do  not  have  identifiable 
transition  funding  virtually  eliminates  the  Air  Force  from  maintaining  and 
advancing  a  broad  range  of  warfighting  capabilities.  The  Assistant  Secretary 
stated  that  certain  technologies  are  developed  but  may  not  transition  immediately, 
and  that  discontinuing  the  efforts  would  place  the  Air  Force  at  risk.  The  Assistant 
Secretary  stated  that  there  have  been  many  requests  for  immediate  or  timely 
technology  capabilities  where  AFRL  was  able  to  provide  previously  developed 
solutions. 

The  Assistant  Secretary  provided  the  status  of  seven  of  the  eight  technologies 
included  in  the  recommendation.  The  Assistant  Secretary  stated  that  the 
technology  transition  will  be  budgeted  in  FY  2006;  the  technology  transitioned  to 
another  development  effort;  additional  desired  capabilities  delayed  the  transition 
but  the  transition  plan  is  in  the  approval  stage;  technology  was  complete  but  not 
funded  for  transition;  technology  is  complete  and  unplanned  Defense  Emergency 
Response  Funds  were  used  to  transition  the  technology;  near-term  transition 
remains  unfunded  but  Warfighter  Rapid  Acquisition  Program  funds  were  being 
sought;  and  further  development  was  needed  before  the  planned  receiver  will  seek 
transition  funds.  The  eighth  technology  pertained  to  the  Targets  Under  Trees, 
scheduled  to  transition  to  the  Electronic  Systems  Center  for  which  no  comments 
were  provided. 

Audit  Response,  The  Assistant  Secretary’s  comments  are  nonresponsive.  The 
intent  of  the  recommendation  was  to  address  transitions  that  were  scheduled  for 
EYs  2003  and  2004.  The  Air  Eorce  needs  to  consider  whether  continued 
expenditure  of  ATD  funds  on  technologies  scheduled  for  near-term  transition  to 
acquisition  programs  is  prudent  if  the  necessary  funds  to  continue  the  required 
development  are  not  available.  Although  AERL  has  no  authority  to  compel 
program  managers  to  identify  transition  funding,  they  do  have  the  responsibility 
to  review  technology  paths  and  plans  for  technology  products  to  ensure  viability, 
to  identify  whether  the  necessary  transition  funds  are  available,  and  to  determine 
whether  continued  expenditure  of  ATD  funds  is  appropriate  when  technology 
transitioning  is  questionable.  One  of  the  objectives  of  the  ATC  process  was  to 
ensure  receiver  funding  sponsorship  for  transitioning.  With  limited  S&T  financial 
resources,  it  is  prudent  to  align  S&T  efforts  to  technologies  that  are  more  likely  to 
result  in  successful  transitions. 
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The  recommendation  focused  on  developing  technologies  that  were  scheduled  for 
transition  in  FYs  2003  and  2004.  The  key  to  transitioning  technology  is  the 
availability  of  sufficient  funds  to  mature  the  technology  through  later  TRLs. 

Funds  to  mature  and  test  the  S&T  efforts  are  needed;  however,  the  budget  cycle 
requires  as  much  as  2  years  of  planning  before  funds  are  available.  Therefore, 
AFRL  and  the  planned  receiver  must  agree  early  and  prevent  funding  lapses 
during  development.  As  indicated  in  the  Assistant  Secretary  comments,  at  least 
half  of  those  technologies  remain  unfunded  or  were  provided  to  the  warfighter 
only  through  unplanned  emergency  funding.  Although  the  recommendation 
addressed  only  eight  near-term  technology  transitions,  the  audit  report  identified 
that  the  AFRL  is  developing  technologies  that  have  a  transitioning  funding 
shortfall  of  at  least  $529  million  if  the  technologies  advance  as  planned.  Many  of 
those  emerging  technologies  will  fall  into  a  waiting  period  where  the  technology 
is  ready  for  transition  but  the  planned  recipient  is  unable  to  receive  it  because  the 
recipient  lacks  transition  funding.  That  waiting  period  could  range  from  2  years 
to  a  much  longer  period,  depending  upon  the  planned  receiver’s  competing 
funding  priorities. 

We  request  the  Assistant  Secretary  to  reconsider  his  position  and  provide 
additional  comments  to  the  final  report. 

4.  Issue  guidance  on  critical  experiments  that  require  the 
estahlishment  of  memorandums  of  agreement  or  understanding,  or 
technology  transition  plans,  and  the  initial  development  of  technology 
readiness  levels  and  exit  criteria. 

Management  Comments,  The  Assistant  Secretary  nonconcurred  because  the 
recommendation  would  “require”  formal  coordination  with  planned  recipients. 
The  Assistant  Secretary  stated  that  if  the  recommendation  was  reworded  to 
“consider”  the  Air  Force  would  concur.  The  Assistant  Secretary  also  stated  that 
AFRL  is  preparing  an  instruction  for  CEs  that  include  most,  if  not  all,  of  the 
recommendations  as  options  and  that  establishing  MOAs,  MOUs,  or  TTPs  as 
appropriate  could  be  useful. 

Audit  Response,  The  Assistant  Secretary’s  comments  are  nonresponsive  to  the 
recommendation.  Critical  experiments  and  ATDs  are  funded  with  advanced 
technology  development  funds  under  the  research,  development,  test,  and 
evaluation  appropriation  and  comprise  53  percent  ($380  million)  of  the  Air 
Force’s  budget  for  FY  2003.  The  Handbook  and  Guide  emphasize  that  successful 
transitions  require  commitment  from  all  parties.  Coordination  and  commitment 
are  also  stressed  in  the  Defense  Systems  Management  College  training  course, 
“Program  Management  for  S&T  Manager,”  STM  301.  We  believe  that  continued 
expenditure  of  large  amounts  of  funds  on  CE  technologies  that  do  not  have 
coordinated  paths  or  plans  for  transitioning  to  acquisition  programs  ignores 
lessons  learned  and  training  on  successful  S&T  transitioning.  Successful 
transitioning  requires  coordination  and  funding  considerations  through  formal 
agreements  with  planned  technology  recipients.  The  Assistant  Secretary’s 
comments  do  not  address  the  intent  of  the  recommendation  because  S&T 
managers  were  always  encouraged  to  develop  MOAs  or  MOUs,  TTPs,  TRLs,  and 
exit  criteria.  Table  1  of  this  report  shows  that,  despite  the  encouragement,  only 
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one  of  the  six  CEs  had  established  basic  coordination  agreements.  Therefore,  we 
still  believe  that  advisory  guidance  is  ineffective  and  the  requirement  to  establish 
these  agreements  will  enhance  the  likelihood  of  technology  transitions  as  well  as 
meet  an  identified  military  need.  We  request  the  Assistant  Secretary  to  reconsider 
his  position  and  provide  additional  comments  to  the  final  report. 

5,  Develop  a  process  to  coordinate  critical  experiment  efforts  with  the 
Air  Force  Major  Commands.  This  process  should  require  the  Air  Force 
Major  Commands’  prioritization  of  and  comments  on  applicable  critical 
experiments  for  consideration  hy  the  Air  Force  Research  Laboratory  in 
determining  whether  to  continue  development. 

Management  Comments,  The  Assistant  Secretary  nonconcurred  because  the 
recommendation  “required”  coordination  with  planned  recipients  for  CEs.  The 
Assistant  Secretary  stated  that  coordination  with  planned  recipients  would  be  too 
manpower  intensive  and  would  be  of  marginal  value  in  making  investment 
decisions.  The  Assistant  Secretary  stated  that  information  on  CEs  is  available  on 
the  ATD  database  to  which  many  Major  Commands  have  access  and  that  the 
AERE  portfolio  is  discussed  during  the  Capabilities  Requirements  Review  and 
Assessment  process. 

Audit  Response,  The  Assistant  Secretary’s  comments  are  nonresponsive.  The 
Assistant  Secretary  believes  there  should  be  less  coordination  for  CEs  than  for 
ATDs  even  though  CEs  compose  53  percent  of  the  AERE  advanced  technology 
development  budget.  The  Office  of  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology  and  Eogistics  issued  the  “Manager’s  Guide  to 
Technology  Transition  in  an  Evolutionary  Acquisition  Environment,”  January  31, 
2003.  The  Manager’s  Guide  states  that  technology  projects  should  be  prioritized 
according  to  the  warfighters’  projected  needs  and  reviewed  by  them  periodically. 
S&T  leaders,  warfighters,  and  the  acquisition  or  sustainment  program  manager 
should  do  the  review  annually,  and  projects  should  be  funded  according  to  the 
priorities  established.  As  a  means  of  encouraging  new  ideas,  all  programs  should 
be  evaluated  for  relevance  and  productivity.  One  way  of  encouraging  ideas  is  to 
eliminate  the  least  productive  projects  annually,  which  will  keep  the  technology 
more  current.  The  collaboration  and  coordination  from  early  interaction  and 
exchange  between  the  technology  and  planned  receiver  communities  permit 
experimenting  with  technology-driven  operational  issues  and  the  early  weeding 
out  of  unattainable  technologies  resulting  in  more  focused  requirements  and 
capabilities. 

Air  Combat  Command  Prioritization.  In  August  2001,  the  Director  of 
Requirements  at  Air  Combat  Command  prioritized  AERE  CEs  to  provide 
feedback  on  technology  programs  that  were  important  to  his  command.  The 
command’s  review  and  prioritization  were  the  result  of  an  action  item  at  an 
Applied  Technology  Council  meeting.  In  providing  the  prioritization  to  AERE, 
the  Director  of  Requirements  stated  that  the  AERE  should  continue  to  work 
together  with  the  command  to  ensure  a  common  understanding  of  CE  priorities. 
During  that  review.  Air  Combat  Command  personnel  reviewed  and  ranked 
121  CEs  and  did  not  comment  on  other  CEs  outside  of  the  Command’s  interest. 
The  Air  Combat  Command’s  request  for  review  of  CEs  illustrates  the  importance 
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that  one  major  command  places  on  the  review  of  emerging  technologies  and  its 
attempt  to  influence  AFRL  investment  decisions. 

AFRL  Database.  Although  the  AFRL  database  has  information  on  the 
CEs  that  could  be  used  to  identify  the  nature  of  the  research,  the  database  does  not 
provide  for  prioritization  and  comment. 

AFRL  Portfolio.  The  Assistant  Secretary  stated  that  the  AFRL  portfolio 
is  discussed  during  the  Capabilities  Requirements  Review  and  Assessment 
process.  The  Capabilities  Requirements  Review  and  Assessment  process 
examines  capabilities  in  Global  Strike,  Homeland  Security,  Global  Response, 
Global  Mobility,  Air  and  Space/Command,  Control,  Intelligence,  Surveillance, 
and  Reconnaissance.  This  process  has  recently  been  initiated  and  still  is  in 
development.  It  is  unclear  how  this  top-level  review  process  will  focus  on 
individual  CEs  and  prioritize  them  in  relation  to  the  overall  concept  of  operations. 

We  continue  to  believe  that  the  review  and  prioritization  of  CEs  by  planned 
recipients  would  provide  valuable  insight  to  AERL  in  directing  its  efforts  towards 
satisfying  identified  military  needs.  With  planned-recipient  endorsement  of 
emerging  CEs,  the  recipient  can  ensure  that  CE  efforts  are  directly  related  to  an 
identified  military  need  and  will  more  likely  transition  to  the  warfighter. 
Therefore,  we  request  the  Assistant  Secretary  to  reconsider  his  position  and 
provide  additional  comments  to  the  final  report. 

6.  Modify  the  Statements  of  Duties  and  Experience  for  science  and 
technology  personnel  to  emphasize  the  need  for  coordination  with  planned 
recipients.  The  Statements  of  Duties  and  Experience  should  specifically 
include  the  requirement  for  all  advanced  technology  demonstration-funded 
efforts  to  establish  working-level  integrated  product  teams  with  all  planned 
recipients,  to  create  integrated  product  team  charters,  to  coordinate  and 
agree  upon  quantitative  metrics  and  key  exit  criteria  with  all  planned 
recipients,  and  to  develop  and  maintain  up-to-date  memorandums  of 
agreement  or  understanding,  or  technology  transition  plans. 

Management  Comments.  The  Assistant  Secretary  nonconcurred.  He  stated  that 
S&T  personnel  work  on  a  variety  of  projects  in  basic,  applied  research,  and 
advanced  technology  development  and  that  it  is  not  appropriate  to  single  out 
ATD-related  work  in  the  Statements  of  Duties  and  Experiences. 

Audit  Response.  The  Assistant  Secretary’s  comments  are  nonresponsive  and 
inconsistent  with  the  guidance  issued  by  OSD  and  comments  provided  by  the 
Army  and  the  Navy  on  a  similar  recommendation.  As  described  in  the 
Background  section  of  this  report,  the  DoD  S&T  Affordability  Task  Eorce  issued 
a  Handbook  that  stresses  the  importance  of  coordination  between  the  research  and 
technology  recipient  communities.  The  Handbook  also  states  that  one  of  the  keys 
to  successful  transitioning  is  implementing  an  S&T  personnel  assessment  process 
that  is  based  on  transitioning  and  affordability,  in  addition  to  individual  technical 
achievements  and  publishing  technical  papers.  Accelerating  the  flow  of 
technology  to  the  warfighter  is  one  of  the  top  priorities  of  the  Under  Secretary  of 
Defense  for  Acquisition,  Technology  and  Eogistics.  Einking  technology 
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transitioning  to  S&T  managers’  performance  would  reinforce  the  DoD  priority  of 
enhancing  the  likelihood  of  providing  advanced  technologies  to  the  warfighter. 

We  made  a  similar  recommendation  to  the  Army  and  Navy  S&T  officials  in  their 
respective  technology  transitioning  reviews,  and  the  Army  and  Navy  officials 
concurred  with  the  recommendation.  Army  and  Navy  S&T  officials  also  manage 
basic  and  applied  research  efforts  as  well  as  advanced  technology  development. 
We  request  the  Assistant  Secretary  to  reconsider  his  position  and  provide 
additional  management  comments  to  the  final  report. 
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Appendix  A.  Scope  and  Methodology 


Work  Performed,  We  examined  30  S&T  teehnologies  at  the  AFRL,  with 
FY  2003  funding  of  $123  million,  to  evaluate  the  management  process  for 
transitioning  successful  technologies  to  the  warfighter.  The  S&T  technologies 
examined  were  funded  with  the  advanced  technology  development  budget  activity 
within  the  Air  Force  research,  development,  test  and  evaluation  appropriation. 

The  30  projects  were  judgmentally  selected,  were  currently  funded,  were  planned 
near  term  transitions,  were  high  dollar  value  budgeted  projects,  and  were  from 
6  of  the  8  AFRL  Directorates  that  conduct  ATD  research  projects.  The  FY  2003 
Appropriation  Bill  provided  about  $718  million  in  Air  Force  advanced  technology 
development  funding.  The  FY  2003  budget  provided  funding  for  62  ATDs, 

129  CEs,  and  additional  congressional  interest  items  valued  at  $170  million. 

We  conducted  interviews  with  S&T  and  acquisition  program  officials,  and 
examined  applicable  key  documentation.  Key  documentation  included  guidance 
advocated  by  the  Deputy  Under  Secretary  of  Defense  for  Science  and 
Technology;  technology  transition  plans;  working-level  IPT  meeting  minutes 
where  available;  memorandums  of  understanding  and  agreement;  acquisition 
program  funding  profiles;  S&T  management  plans;  technology  transition  paths  or 
plans;  and  the  research,  development,  test,  and  evaluation  budget  item 
justification  sheet  (R-2  Exhibit). 

We  performed  this  audit  from  September  2002  through  August  2003  in 
accordance  with  generally  accepted  government  auditing  standards.  We  did  not 
evaluate  the  technical  merits  of  the  S&T  products.  We  limited  the  AERE 
management  control  review  to  the  S&T  management  procedures  and  the 
departments  responsible  for  transitioning  technology  from  the  S&T  community  to 
the  acquisition  program  managers  or  another  technology  area. 

Use  of  Computer-Processed  Data,  We  relied  on  computer-processed  data 
without  performing  tests  of  general  and  application  system  controls  to  confirm  the 
reliability  of  the  data.  We  relied  on  the  Department  of  the  Air  Force  Research 
Eaboratory  Advanced  Technology  Demonstration  (ATD)  database, 
https://techsql.afrl.af.mil/atd/default.asp  and  a  computerized  listing  of  ATDs  and 
CEs  to  represent  the  known  universe  of  S&T  products  in  the  review  of  the 
management  process.  We  validated  the  total  funding  for  EY  2003  on  the 
computerized  listing  to  the  total  funding  under  the  advanced  technology 
development  budget  activity  within  the  Air  Eorce  research,  development,  test,  and 
evaluation  appropriation.  Validating  the  computerized  listing  to  the 
appropriations  was  appropriate  for  this  audit  because  the  audit’s  objective  was  to 
examine  the  overall  management  process  for  technology  transitioning,  not  the 
individual  S&T  products.  Eurther  validation  of  the  computerized  listing  would 
not  change  the  conclusions  in  this  report. 

General  Accounting  Office  High-Risk  Area,  The  General  Accounting  Office 
has  identified  several  high-risk  areas  in  the  DoD.  This  report  provides  coverage 
of  the  Weapon  System  Acquisition  high-risk  area. 
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Management  Control  Program  Review 


DoD  Directive  5010.38,  “Management  Control  (MC)  Program,”  August  26,  1996, 
and  DoD  Instruction  5010.40,  “Management  Control  (MC)  Program  Procedures,” 
August  28,  1996,  require  DoD  organizations  to  implement  a  comprehensive 
system  of  management  controls  that  provides  reasonable  assurance  that  programs 
are  operating  as  intended  and  to  evaluate  the  adequacy  of  the  controls. 

Scope  of  the  Review  of  the  Management  Control  Program,  We  reviewed  the 
adequacy  of  AFRL  management  controls  over  technology  transition.  Specifically 
we  reviewed  AFRL  management  controls  over  the  transition  of  science  and 
technology  projects  funded  with  research,  development,  test,  and  evaluation 
funds;  advanced  technology  development  appropriations.  Because  we  did  not 
identify  a  material  control  weakness,  we  did  not  assess  management’s  self- 
evaluation  because  adequate  management  processes  such  as  the  ATC  and  baseline 
reviews  provide  a  mechanism  for  management  controls.  . 

Adequacy  of  Management  Controls,  The  audit  identified  that  the  Air  Force  had 
established  adequate  controls  over  technology  transition  that  included  a  high-level 
management  oversight  process  to  approve  and  review  proposed  and  ongoing 
technologies. 

Prior  Coverage 


During  the  last  5  years,  the  GAO  has  issued  one  report  and  the  Inspector  General  of 
the  Department  of  Defense  (IG  DoD)  has  issued  three  reports  discussing  the  benefits 
of  adequately  managing  the  challenges  of  transitioning  technologies  to  warfighters. 


GAO 


Report  No.  NSlAD-99-162,  “Best  Practices:  Better  Management  of  Technology 
Development  Can  Improve  Weapon  System  Outcomes,”  July  30,  1999 

IG  DoD 

IG  DoD  Report  No.  D-2003-053,  “Navy  Transition  of  Advanced  Technology 
Programs  to  Military  Applications,”  February  4,  2003 

IG  DoD  Report  No.  D-2002-146,  “The  Defense  Advanced  Research  Projects 
Agency’s  Transition  of  Advanced  Information  Technology  Programs,” 
September  1 1 ,  2002 

IG  DoD  Report  No.  D-2002-107,  “Army  Transition  of  Advanced  Technology 
Programs  to  Military  Applications,”  June  14,  2002 
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Appendix  B.  Technology  Readiness  Levels  and 

Their  Definitions 


The  Deputy  Seeretary  of  Defense  issued  interim  aequisition  guidance  on  October  30, 
2002.  The  interim  guidance  provided  a  matrix  that  lists  technology  readiness  levels  and 
descriptions  from  a  systems  approach  for  both  hardware  and  software  as  shown  below. 


Technology  Readiness  Level 

Description 

1.  Basic  principles  observed  and 
reported. 

Lowest  level  of  technology  readiness.  Scientific  research  begins  to 
be  translated  into  applied  research  and  development.  Examples 
might  include  paper  studies  of  a  technology’s  basic  properties. 

2.  Technology  concept  and/or 
application  formulated. 

Invention  begins.  Once  basic  principles  are  observed,  practical 
applications  can  be  invented.  The  applications  are  speculative  and 
there  may  be  no  proof  or  detailed  analysis  to  support  the 
assumptions.  Examples  are  limited  to  analytic  paper  studies. 

3.  Analytical  and  experimental 
critical  function  and/or  characteristic 
proof  of  concept. 

Active  research  and  development  is  initiated.  This  includes 
analytical  studies  and  laboratory  studies  to  physically  validate 
analytical  predictions  of  separate  elements  of  the  technology. 
Examples  include  components  that  are  not  yet  integrated  or 
representative. 

4.  Component  and/or  breadboard 
validation  in  laboratory 
environment. 

Basic  technological  components  are  integrated  to  establish  that  they 
will  work  together.  This  is  relatively  “low  fidelity”  compared  to  the 
eventual  system.  Examples  include  integration  of  “ad  hoc” 
hardware  in  a  laboratory. 

5.  Component  and/or  breadboard 
validation  in  relevant  environment. 

Fidelity  of  breadboard  technology  increases  significantly.  The 
basic  technological  components  are  integrated  with  reasonably 
realistic  supporting  elements  so  it  can  be  tested  in  simulated 
environment.  Examples  include  “high  fidelity”  laboratory 
integration  of  components. 

6.  Sy stem/ subsystem  model  or 
prototype  demonstration  in  a 
relevant  environment. 

Representative  model  or  prototype  system,  which  is  well  beyond 
that  of  TRL  5,  is  tested  in  a  relevant  environment.  Represents  a 
major  step  up  in  a  technology’s  demonstrated  readiness.  Examples 
include  testing  a  prototype  in  a  high  fidelity  laboratory  environment 
or  in  a  simulated  operational  environment. 

7.  System  prototype  demonstration 
in  an  operational  environment. 

Prototype  near,  or  at,  planned  operational  system.  Represents  a 
major  step  up  from  TRL  6,  requiring  the  demonstration  of  an  actual 
system  prototype  in  an  operational  environment  such  as  an  aircraft, 
vehicle,  or  space.  Examples  include  testing  the  prototype  in  a  test 
bed  aircraft. 

8.  Actual  system  completed  and 
qualified  through  test  and 
demonstration. 

Technology  has  been  proven  to  work  in  its  final  form  and  under 
expected  conditions.  In  almost  all  cases,  this  TRL  represents  the 
end  of  true  system  development.  Examples  include  developmental 
test  and  evaluation  of  the  system  in  its  intended  weapon  system  to 
determine  if  it  meets  design  specifications. 

9.  Actual  system  proven  through 
successful  mission  operations. 

Actual  application  of  the  technology  in  its  final  form  and  under 
mission  conditions,  such  as  those  encountered  in  operational  test 
and  evaluation.  Examples  include  using  the  system  under 
operational  mission  conditions. 
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Advanced  Technology  Demonstrations 


System 

Receiving 

Acquisition 

Program 

MOA/ 

MOU/ 

TTpi 

With 

User 

Established 
an  IPT^ 

Advanced 

Laser  Eye 
Protection 
(ALEP) 

311*  Human  Systems 
Wing  (3 11  HSW) 

Yes 

Yes 

Cognitive 

Desktop 

Information 

Manager 

(CDIM) 

Electronic  Systems 
Center  (ESC) 

n/a^ 

n/a" 

Common 

Situation 

Awareness 

(CSA) 

Aeronautical  Systems 
Center  (ASC) 

No 

Yes 

Day/Night 

EO/IR  Threat 
Tracker 

Countermeasure 

Aeronautical  Systems 
Center  (ASC) 

No 

No 

Distributed 

Analysis 

Decision 

Support 

System 

(DADSS) 

Electronic  Systems 
Center  (ESC) 

n/a^ 

n/a” 

Effects  Based 

Operations 

(EBO) 

Electronic  Systems 
Center  (ESC) 

Draft^ 

Yes 

ipt2 

Charter 

IPT2 

Includes 

Acquisition 

User 

ipt2 

Includes 

Acquisition 

Prime 

Contractor 

Funding 

By 

Acquisition 
User  to 
Transition^ 

Acquisition 
User 
Agreed 
to  TRL^ 

Acquisition 

User 

Agreed 

to 

Exit 

Criteria 

No 

Yes 

Yes 

Yes  (Cat  1) 
(FY03) 

No 

Yes 

n/a^ 

n/a^ 

n/a^ 

n/a^ 

n/a^ 

n/a" 

No 

Yes 

Yes 

n/a'"  (Cat  1) 

No 

No 

n/a'’ 

n/a® 

n/a® 

n/a'" 

(Cat  2A) 

No 

No 

n/a^ 

n/a^ 

n/a^ 

n/a'* 

n/a^ 

n/a" 

No 

Yes 

Yes 

No  (Cat  2A) 
(FY  04) 

No 

No 

Note:  See  footnotes  at  the  end  of  the  appendix. 


ppendix  C.  Summary  of  Advanced  Technology 
Development  Projects  Reviewed 


Advanced  Technology  Demonstrations  (cont’d) 


MOA/ 

MOU/ 

rT-rx-T^l 


System 

Receiving 

Acquisition 

Program 

TTpi 

With 

User 

Established 
an  IPT^ 

Enhanced  Rotor  Life 
Extension  (ERLE) 

Spiral  I 

Oklahoma  City  Air 
Logistics  Center 
(OC-ALC) 

Yes 

Yes 

Expeditionary  Air 
Ground  Equipment 
(EAGE) 

Warner  Robbins  Air 
Logistics  Center 
(WR-ALC) 

Draft^ 

No 

F35F136 

Aeronautical 

Systems  Center 
(ASC) 

Draft^ 

Yes 

Foliage  Penetration 
Synthetic  Aperture 

Radar  (FOPEN) 

Aeronautical 

Systems  Center 
(ASC) 

Draft^ 

Yes 

Future  Air  Navigation 
and  Traffic  Avoidance 

Electronic  Systems 
Center  (ESC) 

No 

No 

Solution  Through 
Integrated 
Communication, 
Navigation  & 
Surveillance  (CNS) 
(FANTASTIC) 


IPT2 

Charter 

IPT2 

Includes 

Acquisition 

User 

IPT2 

Includes 

Acquisition 

Prime 

Contractor 

Funding 

By 

Acquisition 

IJserto 

Transition^ 

Acquisition 
User 
Agreed 
to  TRL^ 

Acquisition 

User 

Agreed 

to 

Exit 

Criteria 

No 

Yes 

Yes 

n/a‘°  (Cat  1) 

No 

No 

n/a^ 

n/a® 

n/a'" 

n/a’ 

No 

No 

No 

Yes 

Yes 

n/a'° 

No 

No 

Yes 

Yes 

Yes 

No  (Cat  2B) 
(FY03) 

No 

No 

n/a^ 

n/a^ 

n/a'" 

n/a’ 

No 

No 

Note:  See  footnotes  at  the  end  of  the  appendix. 


Advanced  Technology  Demonstrations  (cont’d) 


K) 


System 

Reeeiving 

Aequisition 

Program 

MOA/ 

MOU/ 

TTpi 

With 

User 

Established 
an  IPT^ 

Fuze  Air-to-Surfaee 
Teehnology  (FAST) 

Air  Armaments 
Center  (AAC) 

Yes 

Yes 

Global  Air  Mobility 
Advaneed  Teehnologies 
(GAMAT) 

Air  Mobility 
Command  (AMC) 

Draft^ 

Yes 

Information  for  Global 
Reaeh  (IFGR) 

Eleetronie  Systems 
Center  (ESC) 

Draft^ 

Yes 

Integrated  Flight 
Management  (IFM) 

Eleetronie  Systems 
Center  (ESC) 

Draft^ 

Yes 

Integrated  Panoramie 
Night  Vision  Goggles 
(IPNVG) 

Aeronautieal 

Systems  Center 
(ASC) 

Yes 

Yes 

Laser  Infrared  Flyout 
Experiment  (LIFE) 

Aeronautieal 

Systems  Center 
(ASC) 

Yes 

Yes 

ipt2 

Charter 

IPT2 

Ineludes 

Aequisition 

User 

ipt2 

Ineludes 

Aequisition 

Prime 

Contraetor 

Funding 

By 

Aequisition 

LTserto 

Transition 

Aequisition 
User 
Agreed 
to  TRL'* 

Aequisition 

User 

Agreed 

to 

Exit 

Criteria 

No 

Yes 

Yes 

n/a'°  (Cat  2B) 

Yes 

Yes 

No 

Yes 

n/a* 

No  (Cat  2A) 
(FY03) 

No 

No 

No 

Yes 

Yes 

n/a'°  (Cat  2B) 

No 

No 

No 

Yes 

Yes 

Yes  (Cat  1) 
(FY03) 

No 

No 

No 

Yes 

Yes 

Yes  (Cat  1) 
(FY03) 

Yes 

Yes 

No 

Yes 

Yes 

Yes  (Cat  1) 
(FY04) 

No 

yes 

Note:  See  footnotes  at  the  end  of  the  appendix. 


Advanced  Technology  Demonstrations  (cont’d) 


System 

Receiving 

Acquisition 

Program 

MOA/ 

MOU/ 

TTpi 

With 

User 

Established 
an IPT^ 

Multiple  Event  Hard 
Target  Fuze  (MEHTF) 

Air  Armaments 
Center  (AAC) 

Yes 

Yes 

Powered  Low-Cost 
Autonomous  Attack 
System  (P-LOCAAS) 

Air  Armaments 
Center  (AAC) 

Yes 

Yes 

Precision  Direct  Attack 
Munition  (PD AM) 

Air  Armaments 
Center  (AAC) 

Yes 

Yes 

Situational  Awareness 
from  Enhanced  Threat 
Information  (SAFETI) 

Aeronautical 
Systems  Center 
(ASC) 

Yes 

Yes 

Surface  Target  Ordnance 
Package  (STOP) 

Air  Armaments 
Center  (AAC) 

Yes 

Yes 

ipt2 

Charter 

IPT2 

Includes 

Acquisition 

User 

ipt2 

Includes 

Acquisition 

Prime 

Contractor 

Funding 

A  . 

Acquisition 
User  to 
Transition^ 

Acquisition 

User 

Agreed 

toTRL'* 

Acquisition 

User 

Agreed 

to 

Exit 

Criteria 

No 

Yes 

Yes 

No  (Cat  2B) 
(FY02) 

Yes 

Yes 

No 

Yes 

Yes 

n/a'°  (Cat  2B) 

Yes 

Yes 

No 

Yes 

Yes 

Yes  (Cat  1) 
(FY03) 

Yes 

Yes 

No 

Yes 

Yes 

n/a^ 

No 

Yes 

No 

Yes 

Yes 

No  (Cat  2A) 
(FY02) 

Yes 

Yes 

Note:  See  footnotes  at  the  end  of  the  appendix. 


Advanced  Technology  Demonstrations  (cont’d) 


System 

Reeeiving 

Aequisition 

Program 

MOA/ 

MOU/ 

TTpi 

With 

User 

Established 
an  IPT^ 

ipt2 

Charter 

ipt2 

Ineludes 

Aequisition 

User 

IPT2 

Ineludes 

Aequisition 

Prime 

Contraetor 

Funding 

By 

Aequisition 

IJserto 

Transition^ 

Aequisition 
User 
Agreed 
to  TRL^ 

Aequisition 

User 

Agreed 

to 

Exit 

Criteria 

Strike  Helmet  21 
Teehnologies  (SH21) 

Aeronautieal 

Systems  Center 
(ASC) 

Yes 

Yes 

No 

Yes 

Yes 

No  (Cat  2A) 
(FY04) 

Yes 

Yes 

Targets  Under  Trees 
(TUT) 

Aeronautieal 

Systems  Center 
(ASC) 

Draft^ 

Yes 

Draft^ 

Yes 

Yes 

No  (Cat  2B) 
(FY04) 

No 

No 

Targets  Under  Trees 
(TUT)  (eont’d) 

Eleetronie  Systems 
Center  (ESC) 

Draft^ 

Yes 

Draft^ 

No 

No 

No  (Cat  2B) 
(FY04) 

No 

No 

K) 


Note:  See  footnotes  at  the  end  of  the  appendix. 


Critical  Experiments 


MOA/ 

MOU/ 


System 

Reeeiving 

Aequisition 

Program 

TTP' 

With 

User 

Established 
an  IPT^ 

High  Speed  Penetrator 
with  Preeision  Fuzing 
(HSP) 

Air  Armaments 
Center  (AAC) 

No 

No 

Hyperspeetral 
Information  Fusion 
(HSIF) 

Eleetronie  Systems 
Center  (ESC) 

No 

Yes 

Joint  Battlespaee 
Infosphere  (JBI) 

Eleetronie  Systems 
Center  (ESC) 

No 

Yes 

Multi  Intelligenee 

Fusion  for  Time 

Critieal  Targeting 

Eleetronie  Systems 
Center  (ESC) 

No 

No 

Nighttime  Agile  Laser 
Eye  Proteetion 
(NALEP) 

(not  eommissioned) 

No 

Yes 

Programmable 

Integrated  Ordnanee 
Suite  (PIOS) 

Air  Armaments 
Center  (AAC) 

Yes 

Yes 

ipt2 

Charter 

ipt2 

Ineludes 

Aequisition 

User 

ipt2 

Ineludes 

Aequisition 

Prime 

Contraetor 

Funding 

By 

Aequisition 
User  to 
Transition^ 

Aequisition 
User 
Agreed 
to  TRL^ 

Aequisition 

User 

Agreed 

to 

Exit 

Criteria 

n/a^ 

n/a'" 

n/a^ 

n/a'° 

No 

No 

No 

Yes 

Yes 

n/a'° 

No 

No 

No 

Yes 

n/a* 

n/a'° 

No 

No 

n/a^ 

n/a^ 

n/a^ 

n/a'° 

No 

No 

No 

Yes 

Yes 

n/a'° 

No 

No 

Yes 

Yes 

Yes 

n/a*® 

No 

Yes 

Note:  See  footnotes  at  the  end  of  the  appendix. 


*  Memorandum  of  Agreement;  Memorandum  of  Understanding;  Teehnology  Transition  Plan 
^  Integrated  Produet  Team 

^  The  notation  of  “no”  indieates  that  transition  funding  for  the  emerging  teehnology  was  not  speeifieally  identifiable  in  budget  documentation  at  the 
respective  program  office. 

Technology  Readiness  Level 

^  Draft  documentation  was  not  counted  as  a  positive  response. 

^  Not  applicable  because  no  IPT  was  created  for  this  program. 

^  Not  applicable  because  this  technology  was  completed  at  AFRL  prior  to  FY03. 

*  Not  applicable  because  there  is  no  acquisition  prime  contractor  for  this  program. 

^  Due  to  a  transition  within  AFRL,  this  technology  required  no  coordination.  As  such,  all  data  points  are  not  applicable. 

***  Not  applicable  because  budgets  for  FY05  and  beyond  were  not  formalized  at  the  time  of  the  audit. 


K) 

so 


Appendix  D.  Report  Distribution 


Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 
Deputy  Under  Secretary  of  Defense  for  Advanced  Systems  and  Concepts 
Under  Secretary  of  Defense  (Comp troller)/Chief  Financial  Officer 
Deputy  Chief  Financial  Officer 
Deputy  Comptroller  (Program/Budget) 


Department  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  for  Acquisition 

Assistant  Secretary  of  the  Air  Force  (Financial  Management  and  Comptroller) 
Commander,  Air  Force  Materiel  Command 
Commander,  Aeronautical  Systems  Center 
Commander,  Electronic  Systems  Center 
Commander,  Air  Force  Research  Laboratory 
Commander,  Air  Armament  Center 
Commander,  Air  Combat  Command 
Commander,  Air  Force  Special  Operations  Command 

Commander,  Air  Force  Command  and  Control  and  Intelligence,  Surveillance,  and 
Reconnaissance  Center 
Commander,  Air  Mobility  Command 
Auditor  General,  Department  of  the  Air  Force 


Non-Defense  Federal  Organization 

Office  of  Management  and  Budget 
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Congressional  Committees  and  Subcommittees,  Chairman  and 
Ranking  Minority  Member 

Senate  Committee  on  Appropriations 

Senate  Subeommittee  on  Defense,  Committee  on  Appropriations 
Senate  Committee  on  Armed  Services 
Senate  Committee  on  Governmental  Affairs 
House  Committee  on  Appropriations 

House  Subcommittee  on  Defense,  Committee  on  Appropriations 
House  Committee  on  Armed  Services 
House  Committee  on  Government  Reform 

House  Subcommittee  on  Government  Efficiency  and  Financial  Management,  Committee 
on  Government  Reform 

House  Subcommittee  on  National  Security,  Emerging  Threats,  and  International 
Relations,  Committee  on  Government  Reform 
House  Subcommittee  on  Technology,  Information  Policy,  Intergovernmental  Relations, 
and  the  Census,  Committee  on  Government  Reform 
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Department  of  the  Air  Force  Comments 


Final  Report 
Reference 


DEPARTMENT  OF  THE  AIR  FORCE 


WASHINGTON  DC 


Orficv  01  Ths  Af  ilstant  S«cre(ary 


£S  JUl  2003 


MEMORANDUM  FOR  UNDER  ASSISTANT  INSPECTOR  GENERAL  FOR  AUDITING 
OFRCE  OF  THE  INSPECTOR  GENFJi  AL  DEPARTMENT  OF 
DEFENSE 

FROM:  Assistant  Secretary  of  the  Air  Force  (Acquisition)  -  SAF/AQ 

1060  Air  Force  Pentagon 
Washington.  DC  20330-1060 

SUBJECT;  Draft  “Air  Force  Transition  of  Advanced  Technology  Programs  to  Military 
Applications,"  Project  No.  D20O2AB-O128.  dated  2  June  2003. 

Ref;  (a)  Dt^IG  Memo  of  2  Jun  2003 

This  is  in  reply  to  your  memorandum  (ref  (a))  requesting  the  Assistant  Secretary  of  the  Air  Force 
(FinaiKial  Management  and  Comptroller)  provide  Air  Force  Comments  on  the  subject  report. 
The  attachment  provides  Assistant  Secretary  of  the  Air  Force  (Acquisition)  concurrence  or  non- 
concurrence  with  each  rccoinmcndation,  and  comments  as  you  requested. 

The  report  analyzes  management  indicators  that  have  been  suggested  as  best  practices  which 
continue  to  evolve,  but  are  not  mandatory.  The  Air  Force  is  pleased  that  the  audit  identified  that 
the  Air  Force  had  established  adequate  management  controls  over  technology  transition  via  the 
Applied  Technology  Council  and  the  Pnvgram  Baseline  Reviews. 

The  Air  Force  agrees  with  the  audit  concerning  the  rule  of  the  acquisition  program  managers 
supporting  Integrated  Product  Teams,  citing  the  necessity  for  defining  the  relationships  and 
responsibilities  between  the  Science  and  Technology  program  manager  and  the  acquisition 
program  manager. 

The  Air  Force  does  not  agree  that  all  advanced  technology  development  efforts  need  the  same 
level  of  management  controls  and  therefore  makes  a  distinction  between  Advanced  Technology 
Demonstration  programs  and  Critical  Experiments.  h 


marvAr.sambur 

Assist^t  Secretary  of  the  Air  Force 
(Acquisition) 


Attachment; 

Air  Force  Response  to  DoD(IG)  Draft  Report  D2002AB-01 28 
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Final  Report 
Reference 


Air  l-'orcc  Response  to  l)ol)(l(f)  Draft  Report  D20(I2AB-012S 

I  Retuiiinirndaliuiis  iiiid  Rrspuiises:  Hie  subject  report  provides  six  recommendations 
These  recommendations  are  listed  below  with  Air  Force  responses. 

a  Recommeiidation  Require  the  esiabli.shtiirnl  of  inlesraied  pruducl  trams  and 
charters  for  all  advanced  trchiiolujty  drvrIupiiuMit  rlTorts  to  include  representatives 
fruin  the  Air  Force  Major  ('uinmands,  the  candidate  acquisition  prottrani  olTicesIs). 
and  the  acquisition  pru}>rani  ofRcr  prime  contractor's),  where  (Kissible. 

Resixinse  Air  Force  nonconcurs  with  the  recommendation  to  include  all,  advanced 
technology  development  elTorts  in  an  Integrated  Product  Team  (IPT)  type  activity. 
However,  if  the  phrase,  "advanced  technology  develomnent  elTorts."  is  changed  to 
“Advanced  Technolog>'  Denionstiaiion  nioijiains  "  the  Ait  Force  would  concui  with  this 
recommendation 

Non-Advanced  Technology  DenKWistration  6  3  programs,  otherwise  known  as  C'ntical 
Expenments  (CEs),  are  not  sufficiently  technically  mature  for  an  objective  evaluation  by 
Major  ('ommands  (Mj'VJ('OMs)  or  acquisition  program  offices  to  determine  warfighter 
utility  Additionally,  to  do  a  100  iiercent  evaluation  of  the  6  3  program  would  require  a 
massive  aiiKiunt  of  resources  that  are  not  available  for  this  type  of  activity.  Therefore, 
IPTs  are  not  appropriate  for  all  .Advanced  Technology  Development  efforts,  specifically 
CE  elToits  The  .Air  Force  requires  the  establishment  of  IPTs  for  all  Advanced 
Technology  Demonstration  (ATD)  programs  (approximately  50  |>erc«U  of  the  .Advanced 
Tecdinology  Development  funding)  and  encourages  the  consideration  of  IPTs  for  ('Es  but 
does  not  require  them.  Requiring  IPTs  for  CEs  (approximately  300  ('Es  compared  to 
approximately  60  .ATDs)  would  significantly  increase  (afiproximately  five-fold)  the 
manpower  needed  by  Air  Force  Research  Laboiatorv  (.AFRL),  acquisition  program 
offices,  and  M.AJCDMs  to  imjilement  this  recommendation 

For  the  three  ATDs  cited  as  not  having  IPTs 

-  Two  .ATDs  were  completed  in  FY  01 .  There  is  no  IPT  requited  for  closed  .ATDs 

-  The  tliird  .ATD,  Day.''Night  EO  IR  Threat  Tracker  ('ountermeasure.  was 
commissioned  in  early  CY  02  with  a  straw  man  IPT  (Mar  02),  but  there  was  a  contract 
issue  that  was  not  resolved  until  Mai  03  The  IPT  began  work  in  Jun  03 

.All  Force  has  formalized  its  efforts  for  establislnng  IPT  charteis.  The  .AFRL  has 
established  a  foimat  to  facilitate  IPT  chatter  establishment. 

b  Reconiinendation  Establish  procedures  that  require  the  status  of  technology 
transition  plans  between  the  science  and  teclinolosy  manager,  the  .Air  Force  Major 
('oiniiiands.  and  the  candidate  acquisition  prograni  offices  be  a  subject  of  rev  iew  at 
the  .Applied  Technology  ('ouncils. 

Resixinse  Air  Force  concurs  with  this  recommendation. 

-  .AFRL  implemented  this  lecommendation  on  23  Jun  03 
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-  The  Alt  Force  will  foinialiice  tins  process  in  Air  Force  Instruction  (AFl)  61-101 
which  IS  being  revised  Estimated  completion  in  Mar  04 

c.  Recommeiidation  Review  the  terlinulonv  paths  or  plans  for  the  eight  uiiruiided 
science  and  lechnologv  elTorts  that  are  scheduled  to  Iransilion  in  FVs  2003  and  2004. 
and  disconliiiue  development  for  those  that  do  nut  have  forinal  acquisition  program 
support  with  identiried  runding. 

Resixnise  Air  Force  nonconcurs  with  the  recommendation  to  "discontinue  development" 
of  ATDs  that  do  not  have  formal  acquisition  program  supiwrt  with  identified  funding. 

The  Applied  Techmilogv'  Council  (ATC’)  process  was  established  with  dilTereni  ATD 
categories  to  allow  both  technologv  push  b>'  the  technologists  and  technology  pull  liy  the 
M/VJCOMs  and  acquisition  program  managers  A  balance  between  technology  push  and 
technology  pull  is  required  to  maintain  technological  superiority  Eliminating  ('ategory 
2A  (MAJCOM  support,  funding  pending)  and  2B  (M.AJCOM  support,  no  funding)  .\TDs 
virtually  eliminates  the  Air  Force  from  maintaining  and  advancing  a  broad  range  of 
superior  warfighting  capaliihties  Note  An  Force  ATD  Categories  are  sumniari/ed  on 
page  5  of  tile  report 

•As  an  example,  certain  technologies  are  developed,  but  may  not  be  able  to  transition 
immediately  Discontinuing  these  activities  places  .Air  Force  research  elYorts  at  risk  In 
the  past,  there  have  been  many  requests  for  unniediate  or  timely  technology  capabilities 
where  .AFRL  was  able  to  provide  previously  developed  solutions  One  such  item  is  Next 
(leneration  Transparency  If  .AFRL  had  stopped  the  program  because  of  no  identifiable 
transition  path,  these  technologies  would  not  have  been  available  as  immediate  solutions 
for  T-38  and  F-22  problems 

Sev  en  of  the  eight  .ATDs  of  concern  were  reviewed  -  four  were  in  Category  2A  and  three 
were  in  Category  2B  The  eiglith  ATD  (last  entry  on  page  23  of  the  reiiort)  was 
inadequately  identified  to  allow'  for  a  rev  iew.  For  the  seven  .ATDs  reviewed,  all  had  or 
have  sufficient  funding  to  complete  tlie  S&T  efforts 

-  Effects  Based  Operations  (Cat  2A).  Air  Combat  Command  (AC'C)  is  working  with 
the  Electronic  Systems  Center  (ESC)  for  inclusion  of  this  transition  in  the  FA'  06 
POM  .Additionally,  Warfighter  Rapid  .Acquisition  Piogram  funds  are  being  sought  to 
transition  this  technology  Estimated  Completion  1"  Qtr,  FY  04 

Foliage  Penetiation  Synthetic  .Aperture  Radar  (FOPEN,  Cat  2B).  .ATD  was 
decommissioned  and  efforts  were  transitioned  to  the  Target  Under  Trees  (TUT)  ATD 
No  fuitlier  action 

-  (ilobal  .All  Mobility  Adv  anced  Technologies  (CiAM.AT,  Cat  2 A).  .At  the  ret|uest  of 
the  u.sei,  die  program  scope  is  lieing  expanded  to  provide  additional  capability,  which 
will  delay  the  estimated  completion  date.  .Additionally,  transition  of  this  ATD  has 
been  funded,  resulting  m  an  elevation  to  Category  I  This  is  a  software  transition 
that  will  be  done  directly  by  .Air  Mobility  Command  (.AMC).  A  transition  plan  is  in 
the  approval  stages  with  an  estimated  completion  in  the  l“  Qtr,  FY  04. 
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-  Mulliple  Event  Hard  Target  Fuse  (MEHTF,  Cat  2B)  ATD  activities  were  completed 
111  FY  02  Alt  Armament  Center  (AAC)  desires  this  technologv',  but  has  not  identified 
transition  funding  Tins  technology  will  be  used  in  the  High  Speed  Penetrator  CE 

-  Surface  Target  Ordnance  Package  (STOP,  C at  2A)  ATD  activities  were  completed 
and  tiansitioned  b>'  AAC  to  an  acquisition  piogiain  in  FA'  02  Defense  Emergency 
Resjionse  Funds  produced  weafxin  based  on  this  technologv  in  an  accelerated 
program  for  Operations  Iraqi  Fieedom  No  further  action  required 

-  Strike  Helmet  21  Technologies  (SH2I,  Cat  2A)  Category  was  changed  to  2B,  since 
the  Aeronautical  Systems  Center  ( ASC)  F-l  5  System  ftogram  Office  was  unable  to 
fund  this  transition  in  the  short  term  due  to  competing  priorities.  ASC'  still  desires 
this  technology  Additionally,  Waifighter  Rafiid  Acquisition  Program  funds  are  being 
sought  to  tiansition  this  technologv  ATD  will  be  monitored  by  the  ATC  and  should 
be  completed  in  4“  CJtr,  FY  05. 

-  Targets  l.'nder  Trees  (Hrr,  Cat  2B)  The  2000  Air  Force  S&T  Summit  directed 
funding  to  be  shifted  to  this  elTorl  A  portion  of  the  technology,  the  radar,  sliould  be 
completed  m  FA'  04,  but  furthei  concept  development  m  hy[)erspectral  imaging  is 
needed  to  make  this  technology  more  robast  ASC  desires  this  technology,  but  is 
waiting  for  furthei  developments  before  seeking  transition  funding  This  has  been  a 
special  interest  item  foi  the  last  two  Air  Force  C’hiefs  of  Staff.  ATD  will  be 
reevaluate  in  FA'  04  through  the  ATC  process 

d  Recoiiiiiiendation  Issue  Euidance  on  critical  rxperiiiients  that  require  the 
establishnient  of  iiieiiiurandums  of  aereemeiit  or  of  understandiiic.  or  trchtiolu}>y 
transition  plans,  and  the  initial  development  of  technology  readiness  levels  and  exit 
criteria. 

Resixuise  Air  Force  nonconcurs  with  this  recommendation  due  to  the  “requirement"  to 
establish  Memorandums  of  Agreement  (MO.As)  or  Memoiandums  of  Understanding 
(MOlJs),  or  Technology  Transition  Plans  (TTPs).  Establishing  exit  criteria  for  CEs  is 
part  of  managing  an  Adv  anced  Technology  Development  program  If  tlie  phrase, 
"require,"  is  changed  to  "consider,"  the  .Air  Force  would  concur 

.\FRL  IS  drafting  an  in.siruction  for  CEs  that  will  include  most,  if  not  all,  of  the 
recommendations  as  options  Estalilishing  MOAs,  MOl.'s,  or  TTPs,  as  appiopiiate, 
could  be  useful  in  managing  selected  CE  programs.  Estimated  conqiletion  date  I  Jul  04 

e  RecommeiidalKui  Develop  a  process  to  coordinate  critical  rx|ierimrnl  elTorls 
with  the  Air  Force  Major  Coniniands.  'Tliis  process  should  require  the  .Air  Force 
Major  Coiiiiiiaiids'  prioritization  of  and  coininents  on  applicable  critical 
ex|M‘nnients  for  consideralion  by  the  .Air  Force  Research  lalboratory  in  deteriiiiiiing 
whether  to  continue  development. 

Resixuise  .Air  Force  nonconcurs  with  this  recommendation  due  to  the  specification  to 
"require"  M.AJCOMs  to  prioritize  or  comiiKnt  on  afiplicable  CEs  In  some 
ciicumstances  it  would  be  afipropriate  to  get  M.AJCOM  input  on  planned  CEs,  but  it  is 
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not  appiopiiaie  lo  require  MAJC'OMs  to  piiontiju:  a|>plical)le  CEs  This  is  too  iiiaiqxiwer 
intensive  and  tiie  product  would  be  of  marginal  value  in  making  program  investment 
strateg>'  decisions 

Willie  MAJCOMs  are  made  aware  of  CEs,  thev'  are  not  required  to  prioriti/e  or  comment 
on  all  CEs  AFRL  has  alread>‘  begun  to  develop  a  process  to  coordinate  CEs  with  the 
M/VK'OMs  Information  about  all  CEs  is  currently  available  on  the  ATD  database  to 
which  many  M/VICOM  personnel  have  access.  The  AFRL  portfolio  is  being  discussed 
during  the  Capabilities  Requirements  Rev  iew  and  .'\ssessment  process  in  teims  of  how 
technologies  can  fill  AF  ca^iabihty  gaps  Estimated  compleuon  date  I  Jul  05. 

f  ReLnniineiidjluin  Modify  the  Slatrnicnis  of  Duties  and  Esperieiice  for  scienre 
and  lechnulojiy  personnel  to  emphasize  the  need  for  coordination  with  planned 
recipients.  Tlie  Statement  of  Duties  and  F.xperience  should  specincally  include  the 
requirement  for  all  advanced  technoloKy  demonst ration-funded  efTorts  to  establish 
working-level  integrated  product  teams  with  all  planned  recipients,  to  create 
integrated  product  tram  charters,  to  coordinate  and  agm*  upon  quantitative 
metrics  and  key  exit  criteria  with  all  planned  recipients,  and  to  develop  and 
maintain  up-to-date  iiiemorandunis  of  agreement  or  of  understanding,  or 
technology  transition  plans. 

Resixmse  .Air  Force  nonconcurs  with  this  lecommeiidatioii  to  modify'  the  Statements  of 
Duties  and  Experiences  (SDEs)  for  science  and  technology  personnel 

Science  and  Technology  personnel  w'Ork  on  a  variety  of  different  projects  such  as  basic 
research,  applied  research,  cntical  experiments  and  advanced  technology  demonstrations 
It  IS  not  a|)()ro()i  late  to  single  out  adv  anced  technologv  demonstration  related  work  in  the 
SDE 

2  Other  MiHliricatiuns:  In  addition  to  the  Air  Force  responses  to  the  six  recommendations,  the 
following  modifications  are  recommended 

a  Page  1 ,  "Defense  .Acquisition"  Section.  1*'  paiagraph,  line  2  f'hange  text  to  read 
"October  30,  2002  (rmalized  un  12  May  2003).  which 

-  Rea.son  Tlie  "mteiini  guidance"  from  the  DepuK  Secretary  of  Defense  was 
finalized  on  12  May  03  The  report  should  be  updated  to  reference  the  final  EKiD 
Directive 

b  Page  2,  "An  Force  Process"  Sectron,  1“  paragrajih,  line  7  and  Page  3,  “Obiectives" 
Section,  2*^  paragraph,  hne  10.  The  budget  category  6  2  is  mislabeled  as 
"exploratory  technology  development"  and  "exploratory  research."  The  phrases 
should  be  changed  to  ".Applied  Research  " 

-  Reason  Correct  terminology . 

c.  Page  5,  ".All  Force  .Advance  Technology  Program"  Section,  I  “  paragraph,  lines  3-5 
The  sentences  starting  witli  "Since  1997, . .  "  sliould  be  replaced  with  "Since  1997, 
the  .AFRL  organizational  structure  and  the  .Air  Force  S&T  process  has  changed  and 
the  Air  Force  revised  its  .S&T  guidance  to  reflect  those  changes.  The  .Air  Force 
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Page  22 


Page  24 


Page  25 


Page  25 


Polio  Duruincnl  61-1  (coiiiplricil  on  13  June  2003)  updalo  tlie  roles  and 
responsibilities  of  all  AF  organizations  directly  involved  in  managing  AF  S&T  ** 

-  Reason  The  "diaft  guidance"  listed  was  a  revision  to  the  Air  Force  Policy 
Directive  61-1  which  was  finalized  on  13  June  03 

d  Page  12,  "Funding  Requirements  for  All  Near-Teim  Transitions"  Section,  I"* 

paragtapli.  lines  2-6  Add  two  sentences  aflei  the  line  6  sentence  that  ends  with  “  . 
FYs  2003  through  2007  "  These  sentences  sliould  read,  "This  assumes  that  all 
.\TDs  are  successful.  (Note:  Air  Force  has  indicated  llial  some  .ATDs,  altlioueli 
based  oil  potentially  useful  leclinolotty ,  were  not  successful.)" 

-  Reason  The  discussion  of  potential  budget  shortfalls  for  transition  of  .ATDs 
assumes  that  all  .ATDs  will  be  100“  o  successful  Tliere  is  no  guarantee  of  success 
for  anv  Science  and  Technology  program  and.  therefore,  it  is  unnecessary  to 
include  100®  o  of  the  projected  funding  needed  to  transition  all  ,ATDs 

e.  Page  19,  Apjiendix  C,  second  and  sixth  entries  Delete  these  entries 

-  Rea.son  Two  of  Uie  .ATDs  listed  have  no  data  (all  data  points  listed  as  "not 
applicable")  .■Vs  such,  these  entries  provide  no  useful  data 

f  Page  21,  .Apjiendix  C,  "La.ser  Infrared  Flyout  Experiment  (LIFE)"  entry  In  the 
"Acquisition  User  .Agreed  to  Exit  Ciiteiia"  column,  change  "no"  to  "yes  "  If 
iiHKlified,  the  ratio  should  be  10  of  23  in  Table  I  "Exit  ciiteiia  formally  agreed"  (page 
8)  and  the  "Exit  Criteria"  section  (page  1 1 )  of  the  report. 

-  Rea.son  .A  signed  Technology  Transition  Plan  (TTP)  does  exist,  with  signatures 
by  all  parties  between  .Aug  tiirough  Oct  96,  along  with  a  detailed  annex,  which 
includes  all  of  the  technical  performance  measures  needed  to  indicate  exit  criteria 
ATD  was  changed  from  L.AIRf'M  to  LIFE  The  TTP,  which  refers  to  LAIRf’M, 
was  provided  with  an  explanation  to  the  DoDKi  Team.  While  schedule,  cost  and 
methods  of  demonstration  have  changed  since  the  TTP  was  signed,  tlie  exit 
criteria  have  not  In  addition,  the  same  IPT  that  created  the  CY  1996  TTP  is  still 
together  and  continues  to  agree  on  all  aspects  of  exit  criteria  through  all  program 
evolutions 

g  Page  22,  Appendix  C,  P-L(X'.A.AS  entry,  (’hange  the  entry  in  the  ".Acquisition  User 
.Agreed  to  TRL"  entry  to  read  "yes."  If  modified,  the  ratio  should  be  7  of  23  in  Table 
I  "Exit  TRLs  formally  agreed"  (page  8)  and  the  "Techiwlogy  Readiness  Levels" 
section  (page  1 1 )  of  the  report 

-  Rea.son  The  TTP  was  generated  prior  to  establishment  of  TRLs  The  TTP 
describes  activities  that  lead  to  a  TRL  6. 

h  Page  22,  Apjiendix  C,  ".Surface  Target  Ordnance  Package  (STOP)"  entry,  ('hanged 
the  entry  in  the  "Funding  By  .Acquisition  User  to  Transition"  colunm  to  read  "yes  "  If 
ntodified,  the  ratio  should  be  6  of  13  in  Talile  I  "Funding  for  FYs  2003  and  2004 
transitions  by  the  acquisition  recipient"  (page  8)  and  the  "Funding  for  ATDs 
Reviewed"  section  (page  12)  of  the  iqwrt. 

-  Rea.son  Tins  effort  has  transitioned  and  the  product  is  already  m  the  Air  Force 
inventory 
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